XML Sitemaps Generator

Author Topic: Slow crawling  (Read 1391 times)

wiro

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Slow crawling
« on: July 21, 2016, 10:33:33 AM »
Hi,

I'm using unlimited (PHP) sitemap generator. My site is pretty large (more then 50K pages/link). Questions:

1. It tooks 3 hours to generate 1.4MB (still not finish yet) crawl_dump.log file. Is it normal? Seems like it takes forever.

2. How frequent should I cron this crawling process? Every day?

3. I suspect the crawling process slow down my site. Any idea how to minimize this?

4. For this option: "Make a delay between requests, X seconds after each N requests", any idea what number should I put for the X and N ?

Thank you.





XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Slow crawling
« Reply #1 on: July 22, 2016, 05:21:35 AM »
Hello,

1. The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

2. it depends on the site of website and how frequently the pages added on it, it might be daily updated, as well as weekly, for instance.

3,4. You can try to use "Make delay" setting to slow down crawling and avoid overloading server. Exact setting depend on how much slower it needs to be (for instance, 1 second after each 10th request, or 1 second after each 1 request).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2