• Welcome to Sitemap Generator Forum.
 

excluding pages that are in my robots.txt file

Started by andywozhere, March 31, 2010, 05:47:04 PM

Previous topic - Next topic

andywozhere

HI,

I'm sure this is an easy question to answer, but should the sitemap generator exclude files disallowed in the robots.txt file? Files disallowed in my robots.txt file still seem to be generated in the sitemap. Maybe I have missed something in the settings? Or maybe there is something wrong with my robots file.

I'm trying to not include files with the text "view_map" in the URL. Here is an extract from the robots.txt file:

User-Agent: *
Disallow: /*view_map
Allow: /

Cheers,

Andy


andywozhere

Quote from: admin on March 31, 2010, 11:24:49 PM
Hello,

yes, robots.txt is supported. What is an example URL that is still included in sitemap?

[ External links are visible to forum administrators only ] for example.


andywozhere

Updated to v3.0 and it worked perfectly with the robots.txt file.

Thanks!