takes a long time (3 days... or more)

[sagarra]

Hello, the sitemaps crawling for a lot of time. I put timeout in 0 (unlimited) but it is normal?


It is normal to take so long to make the crawling??? I have a server with 12 processors and 47GB RAM..... please help me thank you



Links depth: 4
Current page: si-se-le-solicita-a-la-empresa-cambio-de-puesto-sin-especificar
Pages added to sitemap: 51635
Pages scanned: 55944 (3,740,450.4 KB)
Pages left: 31301 (+ 3314 queued for the next depth level)
Time passed: 37:18:15
Time left: 20:52:19
Memory usage: 81,996.4 Kb
Resuming the last session (last updated: 2014-04-18 13:52:07)

[XML-Sitemaps Support]

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

[sagarra]

yes but all the time is frozen. And I have to restart again all the time.  this is normal?

[XML-Sitemaps Support]

Hello,

depending on server configuration it might be required to run it multiple times until it finishes.

[sagarra]

How the server should be configured?

[XML-Sitemaps Support]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.