XML Sitemaps Generator

Author Topic: crawl keeps hanging  (Read 6418 times)

jojor2004

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
crawl keeps hanging
« on: October 24, 2014, 09:02:20 PM »
Hi, my crawl script hangs all the time, no particular number of pages can be 20,000 or 75,000 - sometimes crawls for 6hrs others 6mins,

please advise

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: crawl keeps hanging
« Reply #1 on: October 25, 2014, 01:49:38 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jojor2004

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: crawl keeps hanging
« Reply #2 on: October 25, 2014, 03:54:52 PM »
that would sound right if it wasn't for the fact that the script has previously created a sitemap with 150k pages.

jojor2004

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: crawl keeps hanging
« Reply #3 on: October 25, 2014, 09:07:16 PM »
memory is set at 512 and execution time 90 seconds - hang iscompletely random - can run for 20k pages or 6 pages and then stops.

please advise

jojor2004

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: crawl keeps hanging
« Reply #4 on: October 26, 2014, 04:45:27 PM »
any chance of some advice on this - current crawl hangs just after 148935 pages  and will not go any further - just stops

jojor2004

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: crawl keeps hanging
« Reply #5 on: October 26, 2014, 04:55:44 PM »
some further info on this is that i have checked the debug output and it hangs every time it tries to saves it's progress.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: crawl keeps hanging
« Reply #6 on: October 27, 2014, 04:43:07 AM »
Hello,

looks like you would need to increase memory_limit even more then, please try 800M.
Also,  in this case I would recommend to run generator in command line if you have ssh access to your server.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2