• Welcome to Sitemap Generator Forum.
 

How To Crawl ONLY New URLs?

Started by info821, March 10, 2009, 01:14:50 AM

Previous topic - Next topic

info821

Hello,

We have a large site. Each time the sitemap crawls it takes about 7-10 hours.

How can we crawl only new pages to add to the sitemap?

Thank you

XML-Sitemaps Support

Hello,

in order to find your new page sitemap generator should crawl other pages, finding the link to new pages.
In case if all your new pages are localized, for instance in news/ folder, you can specify it as starting URL and no URLs "above" that folder will be crawled by generator.