takes a long time (3 days... or more)
« on: April 18, 2014, 07:48:55 PM »
Hello, the sitemaps crawling for a lot of time. I put timeout in 0 (unlimited) but it is normal?


 It is normal to take so long to make the crawling?????? I have a server with 12 processors and 47GB RAM..... please help me thank you



Links depth: 4
Current page: si-se-le-solicita-a-la-empresa-cambio-de-puesto-sin-especificar
Pages added to sitemap: 51635
Pages scanned: 55944 (3,740,450.4 KB)
Pages left: 31301 (+ 3314 queued for the next depth level)
Time passed: 37:18:15
Time left: 20:52:19
Memory usage: 81,996.4 Kb
Resuming the last session (last updated: 2014-04-18 13:52:07)
Re: takes a long time (3 days... or more)
« Reply #1 on: April 19, 2014, 05:18:41 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Re: takes a long time (3 days... or more)
« Reply #2 on: April 22, 2014, 12:45:47 PM »
yes but all the time is frozen. And I have to restart again all the time.  this is normal?
Re: takes a long time (3 days... or more)
« Reply #3 on: April 22, 2014, 01:55:44 PM »
Hello,

depending on server configuration it might be required to run it multiple times until it finishes.
Re: takes a long time (3 days... or more)
« Reply #4 on: April 22, 2014, 02:19:01 PM »
How the server should be configured?
Re: takes a long time (3 days... or more)
« Reply #5 on: April 23, 2014, 02:33:48 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.