XML Sitemaps Generator

Author Topic: Honour robots.txt ?  (Read 8689 times)

allen5

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Honour robots.txt ?
« on: January 06, 2011, 11:42:38 AM »
Hi,

In our site we have disallowed a form component through robots.txt, the script still crawls these pages and adds them to the sitemap.

Is there a way of excluded certain pages or honouring the robots.txt file?

allen5

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Re: Honour robots.txt ?
« Reply #1 on: January 06, 2011, 11:46:20 AM »
I think I found the answer, would I take the exclusion rules such as:

Disallow: /components/

and Add this to "exclude urls". It would be nice if it automatically honoured robots.txt

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Honour robots.txt ?
« Reply #2 on: January 06, 2011, 09:05:10 PM »
Hello,

robots.txt exclusion should work as well.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2