crawling still runs but stuck at a point
« on: October 06, 2011, 11:41:34 PM »

I have two things:

My website has 400k webpages and the crawling stops at 80k and it reattempts many times to get pass that point.  I've look at the forum and it seems I need to increase memory_limit however I spoke to webhost tech support and he said I am not allowed to increase memory_limit.  I don't know if I even need to increase it.  It is set at 128MB.

2.  Since I've managed to get up to 80k, I see sitemap.xml file size is 0.  Does the script writes sitemap.xml during crawling or after it completely crawls 100%?

Re: crawling still runs but stuck at a point
« Reply #1 on: October 07, 2011, 09:15:33 AM »

I managed to convinced tech support to up the memory_limit for only a day and it works...but...after an hour or so...the script started all over!  It says resuming the last session 1970-01-01 00:00:00.  I checked crawl_dump.log and its file size greatly reduced.  "Page added to sitemap" was over 90,000 then drops down to 8,000.  I hope it won't happen again since I only have a day with increased memory_limit.

Also,  sitemap.xml file size is still 0...I've changed permission to 0666 before I started the crawling...or is that to be expected until crawling finishes?

Re: crawling still runs but stuck at a point
« Reply #2 on: October 08, 2011, 09:18:54 AM »

please let me know your generator URL/login in private message to check this.

Sitemap files are created only afterthe crawling is completed.