Slow Crawling
« on: September 14, 2018, 01:16:57 PM »
Links depth: 5
Current page:
Pages added to sitemap: 6259
Pages scanned: 6268 (536,029.1 KB)
Pages left: 1913 (+ 7888 queued for the next depth level)
Time passed: 7:22:16
Time left: 2:14:59
Memory usage: 25,123.0 Kb

Very slow - can we speed this up - thx
Re: Slow Crawling
« Reply #1 on: September 15, 2018, 09:39:08 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In this case looks like pages are loading much slower though.
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Slow Crawling
« Reply #2 on: September 19, 2018, 05:18:47 PM »
Just for statistic. There's something to add: 105,000 dynamically generated pages are scanned by the script about 12 hours, but as I learned, proceeding from the simplest calculations, it's normal.

So, the answer is: it's normal.