XML Sitemaps Generator

Author Topic: how to speed up the crawling speed?  (Read 5769 times)

helmut1

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
how to speed up the crawling speed?
« on: March 03, 2015, 06:18:32 AM »
how to speed up the crawling speed?
he have around 18 million real unique sites and with the actual index speed it would take
nearly 3 month or more.

is there a setting to increase the crawling speed?
i'm thinking like to increase the number or crawlers what visiting and parsing the site simultaneous.
our server resources like ram is not a problem and our multi-core processor can still handle much more threads.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: how to speed up the crawling speed?
« Reply #1 on: March 03, 2015, 06:44:28 AM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2