XML Sitemaps Generator

Author Topic: Using robots.txt file with sitemap.xml  (Read 20332 times)

tjholzk

  • Approved member
  • *
  • Posts: 4
Using robots.txt file with sitemap.xml
« on: March 16, 2008, 06:57:38 PM »
Does anyone know if it is necessary to have the sitemap.xml document in sync with the robots.txt file? Or will the crawlers disregard and not crawl any address that is stated as "disallow" in the robots.txt file, regardless of whether it is listed in the sitemap.xml document or not?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Using robots.txt file with sitemap.xml
« Reply #1 on: March 16, 2008, 08:06:40 PM »
Hello,

oour sitemap generator (both free online and standalone version) support robots.txt protocol and do not include disallowed URLs in sitemap.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

tjholzk

  • Approved member
  • *
  • Posts: 4
Re: Using robots.txt file with sitemap.xml
« Reply #2 on: March 17, 2008, 01:04:56 PM »
Thank you. One more question ... is it necessary to place the robots.txt file in the root directory of the site before running your XML Sitemap generator for that same site?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Using robots.txt file with sitemap.xml
« Reply #3 on: March 17, 2008, 10:45:36 PM »
If you are planning to partially restrict crawling of your site in robots.txt, then you should create that file before generating sitemap so that our sitemap generator can read it and follow the rules you defined.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2