XML Sitemaps Generator

Author Topic: How To Crawl ONLY New URLs?  (Read 14216 times)


  • Registered Customer
  • Approved member
  • *
  • Posts: 2
How To Crawl ONLY New URLs?
« on: March 10, 2009, 01:14:50 AM »

We have a large site. Each time the sitemap crawls it takes about 7-10 hours.

How can we crawl only new pages to add to the sitemap?

Thank you

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10764
Re: How To Crawl ONLY New URLs?
« Reply #1 on: March 10, 2009, 12:31:55 PM »

in order to find your new page sitemap generator should crawl other pages, finding the link to new pages.
In case if all your new pages are localized, for instance in news/ folder, you can specify it as starting URL and no URLs "above" that folder will be crawled by generator.
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.


SMF 2.0.13 | SMF © 2014, Simple Machines