Crawler just churns and churns
« on: July 18, 2011, 09:46:41 PM »
I've upgraded to v5.0 and am running the sitemap generator as we speak but its just churning and churning (for hours) without results.  I "xed" out the browser and tried to log back the browser will never reload the /generator/ page.  I looked on the server and ror.xml and sitemap.xml have not been modified.

Can someone pls help and let me know how I can fix this?!

Thanks.
Re: Crawler just churns and churns
« Reply #1 on: July 19, 2011, 11:47:38 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Re: Crawler just churns and churns
« Reply #2 on: July 19, 2011, 05:30:55 PM »
Hi Oleg,

It finally worked after I left it alone for many hours. I set up a new crontab for the sitemap to recreate during a low activity time to hopefully increase the performance.

Code: [Select]
/usr/local/bin/php /var/www/vhosts/mydomain.com/httpdocs/generator/runcrawl.php


I would prefer to recreate the sitemap every day, but for now I set to once a week to not overload the server.

Any suggestions?
« Last Edit: July 19, 2011, 05:35:04 PM by rshandelman »
Re: Crawler just churns and churns
« Reply #3 on: July 19, 2011, 11:14:58 PM »
You can use the "Make delay for X seconds after each Y request" option - that will slow down the crawler, making the overall generation process longer, but reducing the server load.
Re: Crawler just churns and churns
« Reply #4 on: July 20, 2011, 04:54:16 AM »
ok, cool. I'll try that. thx