crawling still runs but stuck at a point

[admin498]

Hello,

I have two things:

My website has 400k webpages and the crawling stops at 80k and it reattempts many times to get pass that point.  I've look at the forum and it seems I need to increase memory_limit however I spoke to webhost tech support and he said I am not allowed to increase memory_limit.  I don't know if I even need to increase it.  It is set at 128MB.

2.  Since I've managed to get up to 80k, I see sitemap.xml file size is 0.  Does the script writes sitemap.xml during crawling or after it completely crawls 100%?

Thanks!

[admin498]

Hello,

I managed to convinced tech support to up the memory_limit for only a day and it works...but...after an hour or so...the script started all over!  It says resuming the last session 1970-01-01 00:00:00.  I checked crawl_dump.log and its file size greatly reduced.  "Page added to sitemap" was over 90,000 then drops down to 8,000.  I hope it won't happen again since I only have a day with increased memory_limit.

Also,  sitemap.xml file size is still 0...I've changed permission to 0666 before I started the crawling...or is that to be expected until crawling finishes?

Thanks

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.

Sitemap files are created only afterthe crawling is completed.