Sitemap Generator Forum

Links depth: 5
Current page:
Pages added to sitemap: 6259
Pages scanned: 6268 (536,029.1 KB)
Pages left: 1913 (+ 7888 queued for the next depth level)
Time passed: 7:22:16
Time left: 2:14:59
Memory usage: 25,123.0 Kb

Very slow - can we speed this up - thx

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In this case looks like pages are loading much slower though.

Just for statistic. There's something to add: 105,000 dynamically generated pages are scanned by the script about 12 hours, but as I learned, proceeding from the simplest calculations, it's normal.

So, the answer is: it's normal.

We have updated to v.8 on our dev server. Approx. 12.000 pages after 12 hours. Hardware ist excellent.
Production server still v7.2, comparable hardware, identical crawler configuration and pages: all 38.000 pages in 5 hours.
This is not  normal.
Do you have any ideas? 

Hello,

is the configuration the same (generator/data/generator.conf file)?

We have updated to v.8 on our dev server. Approx. 12.000 pages after 12 hours. Hardware ist excellent.
Production server still v7.2, comparable hardware, identical crawler configuration and pages: all 38.000 pages in 5 hours.
This is not  normal.
Do you have any ideas?

Exactly. It was faster before installing v8.0

Hello,

can you please PM me links to both v8.0 instance and older version instance so that I can test it?

Hi,
did you get the link?

Hello,

yes, replied back on Oct 1st, please check PM inbox.