XML Sitemaps Generator

Author Topic: crawl hangs consistently when trying for 5 levels or more  (Read 12840 times)

kgoeller

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
crawl hangs consistently when trying for 5 levels or more
« on: March 25, 2009, 08:46:51 PM »
We're having a problem running site maps greater than 4 levels deep.  The generator consistently hangs with no error message.  I've checked php.ini and upped timeouts and memory allocation.  Currently, we're at 600 seconds for script timeout and parsing timeout, and 64mb memory allocation.

When I run level 4, it completes in less than 2 minutes with no errors.  When i kick off a level 5 crawl, it won't even complete level 4 - stops about halfway through level 4.

Level 4 sitemap has a little over 6100 pages.

We're getting kinda desperate to run a more complete map because we do have some things not indexing properly on Google and we also need to check links at the next levels down.  All help will be gratefully accepted!

Thanks, Karen

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: crawl hangs consistently when trying for 5 levels or more
« Reply #1 on: March 25, 2009, 10:08:30 PM »
Hello,

in most cases it's possible to improve site crawling performance by modifying "Do not parse"/"Exclude" settings. Please PM me your generator link so that I can check it.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

kgoeller

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: crawl hangs consistently when trying for 5 levels or more
« Reply #2 on: March 26, 2009, 05:16:51 PM »
Well, i successfully got a 5-level report run this morning (finally!) and am now trying for level 6.  The ONLY change I made to it was to specify 64MB explicitly in the server memory usage max field on Generator's config tab. That field was previously blank, and it didn't seem to be reading it correctly from the server.

The "do not parse" and "exclude" settings were already fairly well populated, so i left them alone. 

Thanks!
Karen

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: crawl hangs consistently when trying for 5 levels or more
« Reply #3 on: March 27, 2009, 01:22:50 AM »
Replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2