Using robots.txt file with sitemap.xml

[tjholzk]

Does anyone know if it is necessary to have the sitemap.xml document in sync with the robots.txt file? Or will the crawlers disregard and not crawl any address that is stated as "disallow" in the robots.txt file, regardless of whether it is listed in the sitemap.xml document or not?

[admin]

Hello,

oour sitemap generator (both free online and standalone version) support robots.txt protocol and do not include disallowed URLs in sitemap.

[tjholzk]

Thank you. One more question ... is it necessary to place the robots.txt file in the root directory of the site before running your XML Sitemap generator for that same site?

[admin]

If you are planning to partially restrict crawling of your site in robots.txt, then you should create that file before generating sitemap so that our sitemap generator can read it and follow the rules you defined.