How To Crawl ONLY New URLs?
« on: March 10, 2009, 01:14:50 AM »

We have a large site. Each time the sitemap crawls it takes about 7-10 hours.

How can we crawl only new pages to add to the sitemap?

Thank you
Re: How To Crawl ONLY New URLs?
« Reply #1 on: March 10, 2009, 12:31:55 PM »

in order to find your new page sitemap generator should crawl other pages, finding the link to new pages.
In case if all your new pages are localized, for instance in news/ folder, you can specify it as starting URL and no URLs "above" that folder will be crawled by generator.