XML Sitemaps Generator

Author Topic: 3:48:37s processing 84047 pages??  (Read 10735 times)

stefan2

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
3:48:37s processing 84047 pages??
« on: August 13, 2009, 05:13:58 AM »
Hello,

I have the generator installed at [external links are visible to admins only]  . The website is increasing the number of pages from a day to another with aprox 1000 pages. But what is concerning me is the time for processing them. It starts at 3am and finishes after 6am . We don't want the server to be solicited when in the day, but the time increaseas from a day to another...Is this a bug? Is there some option to add only the new pages? Is there other posibility?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: 3:48:37s processing 84047 pages??
« Reply #1 on: August 13, 2009, 09:58:01 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In many cases it is possible to insrease crawling speed with "Do not parse"/"Exclude URLs" options.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2