XML Sitemaps Generator

Author Topic: Stuck Between 800 and 2000 on crawling  (Read 15207 times)

payments69

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Stuck Between 800 and 2000 on crawling
« on: July 15, 2008, 05:06:53 PM »
I am indexing my forum and I got stuck at between 800 2000. I have about 100,000 URLs to crawl and I am sorry but I can't keep on resuming the the crawl since I will be doing that almost all the time. BTW i tried using the solution for this one http://www.xml-sitemaps.com/forum/index.php/topic,1305.html


Godaddy Host:


Here is where it is stuck at

Current page: index.php?topic=10035.30
Pages added to sitemap: 1300
Pages scanned: 1300 (46,462.1 KB)
Pages left: 4438 (+ 4629 queued for the next depth level)
Time passed: 11:23
Time left: 38:54
Memory usage: -

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Stuck Between 800 and 2000 on crawling
« Reply #1 on: July 16, 2008, 06:54:46 AM »
Replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

azmortgages

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Stuck Between 800 and 2000 on crawling
« Reply #2 on: September 17, 2008, 06:47:39 PM »
I just installed 2.9 yesterday in hopes of correcting this same issue. The new version doesn't work any differently. I never get an error msg. It just stops at random.
Links depth: 2
Current page: Art/59897/185/Wedding-Favor-Selection-Some-Considerations.html
Pages added to sitemap: 1463
Pages scanned: 1480 (44,802.2 KB)
Pages left: 36 (+ 7127 queued for the next depth level)
Time passed: 30:28
Time left: 0:44
Memory usage: -
It will stop between 500 and 1500 pages. Restarting means it will crawl another 1000pages +- before it gets stuck again.
I tried at at least 20 times with different settings and I put the IP address in the box at the bottom of the config screen.
Thanks.

azmortgages

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Stuck Between 800 and 2000 on crawling
« Reply #3 on: September 17, 2008, 09:21:49 PM »
All the file permissions are correct. sitemap.xml is empty. Here's where it stopped this time.

Links depth: 2
Current page: Art/60070/186/Visit-the-landmarks-of-Birmingham.html
Pages added to sitemap: 1422
Pages scanned: 1440 (43,630.4 KB)
Pages left: 69 (+ 6950 queued for the next depth level)
Time passed: 29:38
Time left: 1:25
Memory usage: -

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Stuck Between 800 and 2000 on crawling
« Reply #4 on: September 18, 2008, 08:00:51 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2