XML Sitemaps Generator

Author Topic: takes a long time (3 days... or more)  (Read 6855 times)

sagarra

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
takes a long time (3 days... or more)
« on: April 18, 2014, 06:48:55 PM »
Hello, the sitemaps crawling for a lot of time. I put timeout in 0 (unlimited) but it is normal?


 It is normal to take so long to make the crawling?????? I have a server with 12 processors and 47GB RAM..... please help me thank you



Links depth: 4
Current page: si-se-le-solicita-a-la-empresa-cambio-de-puesto-sin-especificar
Pages added to sitemap: 51635
Pages scanned: 55944 (3,740,450.4 KB)
Pages left: 31301 (+ 3314 queued for the next depth level)
Time passed: 37:18:15
Time left: 20:52:19
Memory usage: 81,996.4 Kb
Resuming the last session (last updated: 2014-04-18 13:52:07)

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: takes a long time (3 days... or more)
« Reply #1 on: April 19, 2014, 04:18:41 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sagarra

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: takes a long time (3 days... or more)
« Reply #2 on: April 22, 2014, 11:45:47 AM »
yes but all the time is frozen. And I have to restart again all the time.  this is normal?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: takes a long time (3 days... or more)
« Reply #3 on: April 22, 2014, 12:55:44 PM »
Hello,

depending on server configuration it might be required to run it multiple times until it finishes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sagarra

  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Re: takes a long time (3 days... or more)
« Reply #4 on: April 22, 2014, 01:19:01 PM »
How the server should be configured?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: takes a long time (3 days... or more)
« Reply #5 on: April 23, 2014, 01:33:48 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2