Honour robots.txt ?

[allen5]

Hi,

In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.

Is there a way of excluded certain pages or honouring the robots.txt file?

[allen5]

I think I found the answer, would I take the exclusion rules such as:

Disallow: /components/

and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt

[XML-Sitemaps Support]

Hello,

robots.txt exclusion should work as well.