Sitemap Generator Forum

Hello, the sitemaps crawling for a lot of time. I put timeout in 0 (unlimited) but it is normal?


 It is normal to take so long to make the crawling??? I have a server with 12 processors and 47GB RAM..... please help me thank you



Links depth: 4
Current page: si-se-le-solicita-a-la-empresa-cambio-de-puesto-sin-especificar
Pages added to sitemap: 51635
Pages scanned: 55944 (3,740,450.4 KB)
Pages left: 31301 (+ 3314 queued for the next depth level)
Time passed: 37:18:15
Time left: 20:52:19
Memory usage: 81,996.4 Kb
Resuming the last session (last updated: 2014-04-18 13:52:07)

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

yes but all the time is frozen. And I have to restart again all the time.  this is normal?

Hello,

depending on server configuration it might be required to run it multiple times until it finishes.

How the server should be configured?

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.