XML Sitemaps Generator

Author Topic: Arrghh!! Why is my crawl stopping? It cannot finish!  (Read 18891 times)

phillopez

  • Approved member
  • *
  • Posts: 2
Arrghh!! Why is my crawl stopping? It cannot finish!
« on: August 21, 2008, 09:00:04 PM »
I have restarted the crawl several times already.  I tried both checking the box "running in the background" and unchecked.  It still does not finish.  It just stopped at 2002 pages with "Pages left: 292 (+ 8702 queued for the next depth level)".

I have also tried adding the .htaccess  file as suggested on another post.  When I add this file though, the page comes up as "500 internal server error".

Your immediate assistance is highly appreciated.
 

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #1 on: August 22, 2008, 12:12:58 AM »
Please let me know your generator URL to check the issue.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

phillopez

  • Approved member
  • *
  • Posts: 2
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #2 on: August 22, 2008, 01:23:36 AM »
How do I send you the URL?  Thanks.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #3 on: August 22, 2008, 08:54:07 PM »
Send me a PM: http://www.xml-sitemaps.com/forum/index.php?action=pm;sa=send;u=1 (you should use forums username/password sent to you with sitemap generator download link to be able to send PMs).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

flo

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #4 on: August 25, 2008, 06:09:38 PM »
Report finally generated!  Thanks!!!

flo

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #5 on: August 27, 2008, 05:50:37 AM »
Ooopppsss!  There it goes again.  I added products to my website and tried to crawl it again and it stopped 1/3 along the way.  I did not change any of the settings that you have made.  Please help!!!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #6 on: August 28, 2008, 08:45:07 PM »
How many URLs did it crawl before it stopped and how many URLs you got in sitemap before?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

flo

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #7 on: August 29, 2008, 04:00:12 AM »
I had 7890 when you finished the crawl for me.  Now it has "URLs added: 5739, estimated URLs left: 8907".  I added a lot of products as you can see.  Please assist.  Thanks.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #8 on: August 29, 2008, 07:07:06 PM »
Replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

swof21

  • Approved member
  • *
  • Posts: 1
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #9 on: March 31, 2010, 09:50:56 PM »
Hi, I'm getting similar problems to this. It gets so far and just stops with Error 500. I've tried setting high memory_limit and max_execution_time's but no joy. Can you take a look please: [external links are visible to admins only]

Cheers

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Arrghh!! Why is my crawl stopping? It cannot finish!
« Reply #10 on: March 31, 2010, 10:17:22 PM »
Hello,

in case if you are running on IIS, it might be required to update script timeout as described in http://www.iisadmin.co.uk/?p=7
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2