Arrghh!! Why is my crawl stopping? It cannot finish!

[phillopez]

I have restarted the crawl several times already.  I tried both checking the box "running in the background" and unchecked.  It still does not finish.  It just stopped at 2002 pages with "Pages left: 292 (+ 8702 queued for the next depth level)".

I have also tried adding the .htaccess  file as suggested on another post.  When I add this file though, the page comes up as "500 internal server error".

Your immediate assistance is highly appreciated.

[XML-Sitemaps Support]

Please let me know your generator URL to check the issue.

[phillopez]

How do I send you the URL?  Thanks.

[XML-Sitemaps Support]

Send me a PM: https://www.xml-sitemaps.com/forum/index.php?action=pm;sa=send;u=1 (you should use forums username/password sent to you with sitemap generator download link to be able to send PMs).

[flo]

Report finally generated!  Thanks!!!

[flo]

Ooopppsss!  There it goes again.  I added products to my website and tried to crawl it again and it stopped 1/3 along the way.  I did not change any of the settings that you have made.  Please help!!!

[XML-Sitemaps Support]

How many URLs did it crawl before it stopped and how many URLs you got in sitemap before?

[flo]

I had 7890 when you finished the crawl for me.  Now it has "URLs added: 5739, estimated URLs left: 8907".  I added a lot of products as you can see.  Please assist.  Thanks.

[XML-Sitemaps Support]

Replied to your PM.

[swof21]

Hi, I'm getting similar problems to this. It gets so far and just stops with Error 500. I've tried setting high memory_limit and max_execution_time's but no joy. Can you take a look please: [ External links are visible to forum administrators only ]

Cheers

[XML-Sitemaps Support]

Hello,

in case if you are running on IIS, it might be required to update script timeout as described in [ External links are visible to logged in users only ]