Hi, we had the crawl running well for several days.  It stopped and this time, when we tried to relaunch the crawler, it would never start again.  I believe the first couple times I saw an error message about couldn't open stream something.  But now, I just get a page not found after long time, and the crawl doesn't continue.  Any suggestions on how to get the crawl to continue would be greatly appreciated!

Thanks
Re: Trying to restart, but it is timing out and providing an error page
« Reply #1 on: April 09, 2006, 03:56:11 AM »
Looks like I got it running again!  I rebooted our server and changed the configuration to wait 3 seconds for every 5 requests.  That seems to have gotten it going again!  I don't know if that delay is too long or too short, but seems to be working ok.
Re: Trying to restart, but it is timing out and providing an error page
« Reply #2 on: April 09, 2006, 01:18:07 PM »
Hello,

ok, great.
You can try to configure a bit shorter delays though (or increase the number of requests between delays).
Re: Trying to restart, but it is timing out and providing an error page
« Reply #3 on: April 12, 2006, 11:17:59 PM »
We had to reboot our server, and now we can't get the process started, getting the error messages below:

Resuming the last session (last updated: 2006-04-12 02:17:50)
Warning: fopen(/home/eproxim/public_html/XML-Sitemap-Generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 15

Warning: fwrite(): supplied argument is not a valid stream resource in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 16

Warning: fclose(): supplied argument is not a valid stream resource in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 17

Warning: fopen(/home/eproxim/public_html/XML-Sitemap-Generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 15

Warning: fwrite(): supplied argument is not a valid stream resource in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 16

Warning: fclose(): supplied argument is not a valid stream resource in /home/eproxim/public_html/XML-Sitemap-Generator/pages/class.utils.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 17

Any ideas?  Thanks.
Re: Trying to restart, but it is timing out and providing an error page
« Reply #4 on: April 14, 2006, 04:25:49 AM »
Still running into this problem, anyone else having this issue?

ash

*
  • *
  • 2
Re: Trying to restart, but it is timing out and providing an error page
« Reply #5 on: April 14, 2006, 11:03:18 AM »
Check that the data directory is writeable.

I don't have a permissions problem but I got
Quote
Warning: gzfile(/home/crm/public_html/generator/data/sitemap.xml.gz): failed to open stream: No such file or directory in /home/crm/public_html/generator/pages/page-view.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 34

Warning: implode(): Bad arguments. in /home/crm/public_html/generator/pages/page-view.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 34

I left Oleg a PM as advised in another thread.
Re: Trying to restart, but it is timing out and providing an error page
« Reply #6 on: April 14, 2006, 04:50:41 PM »
Hello,

if you are getting the permissions error message, you should make sure that not only data/ folder has 0777 permissions set, but also all files in data/ folder are set to 0666.