Sitemap Generator Forum

how to speed up the crawling speed?
he have around 18 million real unique sites and with the actual index speed it would take
nearly 3 month or more.

is there a setting to increase the crawling speed?
i'm thinking like to increase the number or crawlers what visiting and parsing the site simultaneous.
our server resources like ram is not a problem and our multi-core processor can still handle much more threads.

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.