Crawler just churns and churns

[rshandelman]

I've upgraded to v5.0 and am running the sitemap generator as we speak but its just churning and churning (for hours) without results.  I "xed" out the browser and tried to log back the browser will never reload the /generator/ page.  I looked on the server and ror.xml and sitemap.xml have not been modified.

Can someone pls help and let me know how I can fix this?!

Thanks.

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.

[rshandelman]

Hi Oleg,

It finally worked after I left it alone for many hours. I set up a new crontab for the sitemap to recreate during a low activity time to hopefully increase the performance.

Code Select Expand

/usr/local/bin/php /var/www/vhosts/mydomain.com/httpdocs/generator/runcrawl.php


I would prefer to recreate the sitemap every day, but for now I set to once a week to not overload the server.

Any suggestions?

[XML-Sitemaps Support]

You can use the "Make delay for X seconds after each Y request" option - that will slow down the crawler, making the overall generation process longer, but reducing the server load.

[rshandelman]

ok, cool. I'll try that. thx