XML Sitemaps Generator

Author Topic: excluding pages that are in my robots.txt file  (Read 8560 times)

andywozhere

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
excluding pages that are in my robots.txt file
« on: March 31, 2010, 04:47:04 PM »
HI,

I'm sure this is an easy question to answer, but should the sitemap generator exclude files disallowed in the robots.txt file? Files disallowed in my robots.txt file still seem to be generated in the sitemap. Maybe I have missed something in the settings? Or maybe there is something wrong with my robots file.

I'm trying to not include files with the text "view_map" in the URL. Here is an extract from the robots.txt file:

User-Agent: *
Disallow: /*view_map
Allow: /

Cheers,

Andy

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: excluding pages that are in my robots.txt file
« Reply #1 on: March 31, 2010, 10:24:49 PM »
Hello,

yes, robots.txt is supported. What is an example URL that is still included in sitemap?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

andywozhere

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: excluding pages that are in my robots.txt file
« Reply #2 on: April 01, 2010, 08:26:57 AM »
[external links are visible to admins only]
Hello,

yes, robots.txt is supported. What is an example URL that is still included in sitemap?

[external links are visible to admins only] for example.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: excluding pages that are in my robots.txt file
« Reply #3 on: April 01, 2010, 07:56:44 PM »
I see that you are running sitemap generator v2.8, please try installing the latest version (3.0)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

andywozhere

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: excluding pages that are in my robots.txt file
« Reply #4 on: April 02, 2010, 10:02:33 AM »
Updated to v3.0 and it worked perfectly with the robots.txt file.

Thanks!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: excluding pages that are in my robots.txt file
« Reply #5 on: April 02, 2010, 01:46:02 PM »
Great!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2