Honour robots.txt ?
« on: January 06, 2011, 11:42:38 AM »
Hi,

In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.

Is there a way of excluded certain pages or honouring the robots.txt file?
Re: Honour robots.txt ?
« Reply #1 on: January 06, 2011, 11:46:20 AM »
I think I found the answer, would I take the exclusion rules such as:

Disallow: /components/

and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt