Sitemap generation crawling problems
« on: June 26, 2006, 04:59:16 PM »
Everything worked fine and I was running the generator through a cron job. Today I noticed that 5 days ago the script did stop generating sitemaps. I ran it manually and it hangs up at the step shown below:

Please wait. Sitemap generation in progress...
Links depth: 3
Current page: kb/article.php?id=032
Pages added to sitemap: 95
Pages scanned: 240 (2,327.8 Kb)
Pages left: 15 (+ 209 queued for the next depth level)
Time passed: 0:13
Time left: 0:00
Memory usage: 571.7 Kb


Looking a t the apache log files the last entry is:

129.42.208.182 - - [26/Jun/2006:11:51:28 -0400] "GET /map/index.php?op=crawlproc&bg=&resume= HTTP/1.1" 200 1812 "[ External links are visible to forum administrators only ]" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"

I'm not sure what to do next?  Any ideas?

Thanks   ... Udo
Re: Sitemap generation crawling problems
« Reply #1 on: June 27, 2006, 10:55:58 PM »
Hello Udo,

when you execute sitemap generator from web interface, the script is usually running with different settings than running as cron job and the maximum execution time is often limited. So, you see the screen stops updating after some time.
Please run sitemap generator from the SSH command line instead (yu can use the same command as specified for cron job) and see how it goes.