Long run time.
« on: July 20, 2018, 03:01:59 PM »
Generator takes 18 hours to generate 70,000 URLs.
This seems very excessive. Google SitemapGenerator would run all 500,000 urls in an hour or so.

We own a dedicated server with 4 multi-core processors and 32GB RAM, but I still can't have this running constantly..

What settings should I check to fix this?

Re: Long run time.
« Reply #1 on: July 21, 2018, 06:44:58 AM »

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.