XML Sitemaps Generator

Author Topic: Crawler gets stuck?  (Read 40508 times)

jon

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Crawler gets stuck?
« on: November 20, 2005, 06:20:44 AM »
I'm not sure if this is normal but I've tried to run this twice now and each time I get to around 5,000 pages the crawler seems to get stuck.  I've checked back on it several times over the last hour and it's at the same number of entries.  Is there any way to tell if it is stuck and if so, what should I do?

Thanks!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Crawler gets stuck?
« Reply #1 on: November 20, 2005, 10:47:12 AM »
Jon,


try to enable "Run in background" checkbox when you run the crawler.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jon

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Crawler gets stuck?
« Reply #2 on: November 20, 2005, 06:31:36 PM »
I did that.  Any other suggestions?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Crawler gets stuck?
« Reply #3 on: November 20, 2005, 07:49:34 PM »
If it is still interrupted, you can use "Save the script state, every X seconds" option to resume crawling after it is stopped. In this case after the script is interrupted, you will see a "Resume scan" checkbox on "Crawling" page.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jon

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Crawler gets stuck?
« Reply #4 on: November 20, 2005, 07:56:14 PM »
OK, I had that set that but I had it at "3600 seconds" and wasn't getting that far. :)

I have noticed that a couple of the times it got stuck on the same URL.  Is there a way to manually bypass potentially troublesome URLs?

jon

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Crawler gets stuck?
« Reply #5 on: November 20, 2005, 08:28:46 PM »
I did bump this down to 600 seconds but I'm not seeing any link to resume the crawl on the crawling page.  That's where I should see it right?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Crawler gets stuck?
« Reply #6 on: November 21, 2005, 10:42:35 AM »
Quote
I have noticed that a couple of the times it got stuck on the same URL.  Is there a way to manually bypass potentially troublesome URLs?
There are special settings on the Configuration page to exclude URLs from being parsed.

Quote
I did bump this down to 600 seconds but I'm not seeing any link to resume the crawl on the crawling page.  That's where I should see it right?
You should try the value of 60 (or even 30 seconds) for this option.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jon

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: Crawler gets stuck?
« Reply #7 on: November 23, 2005, 06:55:26 AM »
I'm still having problems here.  I've tried excluding URLs but that doesn't seem to help.  Any suggestions for next steps?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Crawler gets stuck?
« Reply #8 on: November 23, 2005, 07:49:07 PM »
You should either change your php configuration (increase max_execution_time value in php.ini) or use "Save state"/resume session option.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2