Honour robots.txt ?
« on: January 06, 2011, 11:42:38 AM »
Hi,

In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.

Is there a way of excluded certain pages or honouring the robots.txt file?
Re: Honour robots.txt ?
« Reply #1 on: January 06, 2011, 11:46:20 AM »
I think I found the answer, would I take the exclusion rules such as:

Disallow: /components/

and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt
Re: Honour robots.txt ?
« Reply #2 on: January 06, 2011, 09:05:10 PM »
Hello,

robots.txt exclusion should work as well.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.