• Welcome to Sitemap Generator Forum.
 

crawler problem

Started by saicom2, August 24, 2014, 09:45:59 AM

Previous topic - Next topic

saicom2

Crawler is very slow

pages added in sitemap 2800 in 1 hour

can you help

Regards

XML-Sitemaps Support

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.