XML Sitemaps Generator

Author Topic: Question - Will SiteMap Generator re-crawl the same pages?  (Read 9523 times)

webmaster267

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Question - Will SiteMap Generator re-crawl the same pages?
« on: March 03, 2010, 03:11:24 AM »
Hello,

Beginner question.

I purchased the Sitemap generator, installed and ran it.  Over 40,000 pages were crawled.

Now my question is, when I run the crawl again, will it go over the 40,000 pages again?  

Because those 40,000 + are permanent, they won't be changed.  I  have a folder called "news_of_2010" and that is the only folder updated daily with new links.

Will Google still crawl the old pages? Will Sitemap run over the same pages over and over?
How do I make it only crawl the new folde rthat's updated daily, since the other 40,000 + files are permanent and non-editable.

Thanks in advance.

« Last Edit: March 03, 2010, 03:13:48 AM by webmaster267 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Question - Will SiteMap Generator re-crawl the same pages?
« Reply #1 on: March 04, 2010, 12:51:33 AM »
Hello,

you can keep existing sitemap unchanged, and create a second sitemap for that folder with new files (change sitemap filename and Starting URL accordingly)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2