XML Sitemaps Generator

Author Topic: Crawling process stalls out  (Read 7467 times)

brian47

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Crawling process stalls out
« on: August 22, 2013, 03:51:35 PM »
I've been trying to create a sitemap for the past week but the crawling process never fully gets completed (even if I opt to crawl in the background). I have tried to adjust the php.ini settings but have not seen any improvement. Any guidance will be much appreciated.


Already in progress. Current process state is displayed:

Links depth: 2
Current page: ftr-8519p-5a/
Pages added to sitemap: 895
Pages scanned: 1003 (78,045.7 KB)
Pages left: 1172 (+ 1433 queued for the next depth level)
Time passed: 0:24:27
Time left: 0:28:34
Memory usage: 5,158.1 Kb

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling process stalls out
« Reply #1 on: August 25, 2013, 07:12:58 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

simoneandson

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Re: Crawling process stalls out
« Reply #2 on: September 14, 2013, 06:32:32 PM »
I am having this same issue.  I've increased the memory_limit and max_execution_time and no change.  It seems to fail once the Memory usage gets to around 3,000kb.  memory limit is set to 64mb.  I thought the issue may be the site file path as the hint shows up as C:\Inetpub\...\...\httpdocs\generator/ whereas the document root i believe is just /httpsdocs/sitemap.xml

i've tried multiple variants of the file path and no change in behavior

Links depth: 2
Current page: viewitem.asp?idCategory=52&idProduct=359&priceRange=0x999999
Pages added to sitemap: 443
Pages scanned: 500 (9,114.5 KB)
Pages left: 168 (+ 2274 queued for the next depth level)
Time passed: 0:01:34
Time left: 0:00:31
Memory usage: 3,004.4 Kb

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling process stalls out
« Reply #3 on: September 15, 2013, 03:35:31 PM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
http://forums.iis.net/p/1076662/1585343.aspx#1585343
http://www.iisadmin.co.uk/?p=7
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

brian47

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Crawling process stalls out
« Reply #4 on: September 16, 2013, 08:48:04 PM »
Here was the reply from HostGator:

"Thank you for contacting HostGator. We appreciate your patience and apologize for any inconvenience. Unfortunately, we would be unable to modify the memory limit or time limit for a process to execute. We have these listed in a special support article in addition to all other directives which are unavailable for modification."

I suppose I should have checked compatibility with HostGator prior to purchasing this Sitemap software

randy

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 30
Re: Crawling process stalls out
« Reply #5 on: October 08, 2013, 02:55:34 PM »
Could he use ini_set in the script or increase php_value in .htaccess?

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2