Slow Crawling

[info3294]

Links depth: 5
Current page:
Pages added to sitemap: 6259
Pages scanned: 6268 (536,029.1 KB)
Pages left: 1913 (+ 7888 queued for the next depth level)
Time passed: 7:22:16
Time left: 2:14:59
Memory usage: 25,123.0 Kb

Very slow - can we speed this up - thx

[XML-Sitemaps Support]

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In this case looks like pages are loading much slower though.

[motogonki2]

Just for statistic. There's something to add: 105,000 dynamically generated pages are scanned by the script about 12 hours, but as I learned, proceeding from the simplest calculations, it's normal.

So, the answer is: it's normal.

[psyche]

We have updated to v.8 on our dev server. Approx. 12.000 pages after 12 hours. Hardware ist excellent.
Production server still v7.2, comparable hardware, identical crawler configuration and pages: all 38.000 pages in 5 hours.
This is not  normal.
Do you have any ideas? 

[XML-Sitemaps Support]

Hello,

is the configuration the same (generator/data/generator.conf file)?

[motogonki2]

We have updated to v.8 on our dev server. Approx. 12.000 pages after 12 hours. Hardware ist excellent.
Production server still v7.2, comparable hardware, identical crawler configuration and pages: all 38.000 pages in 5 hours.
This is not  normal.
Do you have any ideas?

Exactly. It was faster before installing v8.0

[XML-Sitemaps Support]

Hello,

can you please PM me links to both v8.0 instance and older version instance so that I can test it?

[psyche]

Hi,
did you get the link?

[XML-Sitemaps Support]

Hello,

yes, replied back on Oct 1st, please check PM inbox.