XML Sitemaps Generator

Author Topic: crawling still runs but stuck at a point  (Read 7915 times)

admin498

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
crawling still runs but stuck at a point
« on: October 06, 2011, 10:41:34 PM »
Hello,

I have two things:

My website has 400k webpages and the crawling stops at 80k and it reattempts many times to get pass that point.  I've look at the forum and it seems I need to increase memory_limit however I spoke to webhost tech support and he said I am not allowed to increase memory_limit.  I don't know if I even need to increase it.  It is set at 128MB.

2.  Since I've managed to get up to 80k, I see sitemap.xml file size is 0.  Does the script writes sitemap.xml during crawling or after it completely crawls 100%?

Thanks!

admin498

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: crawling still runs but stuck at a point
« Reply #1 on: October 07, 2011, 08:15:33 AM »
Hello,

I managed to convinced tech support to up the memory_limit for only a day and it works...but...after an hour or so...the script started all over!  It says resuming the last session 1970-01-01 00:00:00.  I checked crawl_dump.log and its file size greatly reduced.  "Page added to sitemap" was over 90,000 then drops down to 8,000.  I hope it won't happen again since I only have a day with increased memory_limit.

Also,  sitemap.xml file size is still 0...I've changed permission to 0666 before I started the crawling...or is that to be expected until crawling finishes?

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: crawling still runs but stuck at a point
« Reply #2 on: October 08, 2011, 08:18:54 AM »
Hello,

please let me know your generator URL/login in private message to check this.

Sitemap files are created only afterthe crawling is completed.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2