3:48:37s processing 84047 pages??

[stefan2]

Hello,

I have the generator installed at www.tocmai.ro  . The website is increasing the number of pages from a day to another with aprox 1000 pages. But what is concerning me is the time for processing them. It starts at 3am and finishes after 6am . We don't want the server to be solicited when in the day, but the time increaseas from a day to another...Is this a bug? Is there some option to add only the new pages? Is there other posibility?

[XML-Sitemaps Support]

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In many cases it is possible to insrease crawling speed with "Do not parse"/"Exclude URLs" options.