Toggle navigation
PRO Sitemaps
Standalone Generator
About Sitemaps
News
SEO Tools
Sitemap Generator Forum
Guest
Login
Register
Menu
Home
Help
Search
Login
Register
XML Sitemaps Discussions Category
/
Unlimited PHP Sitemap Generator
/
Honour robots.txt ?
Keyword suggestion tool:
Pick the right keywords to target in your SEO and PPC campaigns
« previous
next »
Pages:
1
allen5
11
Honour robots.txt ?
«
on:
January 06, 2011, 11:42:38 AM »
Hi,
In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.
Is there a way of excluded certain pages or honouring the robots.txt file?
Logged
allen5
11
Re: Honour robots.txt ?
«
Reply #1 on:
January 06, 2011, 11:46:20 AM »
I think I found the answer, would I take the exclusion rules such as:
Disallow: /components/
and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt
Logged
XML-Sitemaps Support
11767
Re: Honour robots.txt ?
«
Reply #2 on:
January 06, 2011, 09:05:10 PM »
Hello,
robots.txt exclusion should work as well.
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
Pages:
1
« previous
next »