meta nofollow but in sitemap
« on: July 07, 2010, 09:34:58 PM »
hi!

all my pages i don't want to be in the sitemap.xml have a
<meta name="robots" content="noindex,nofollow" />
tag

but some of them are in the sitemap.xml.
is there something wrong with the meta tag or how does this work?
thanks
mat
Re: meta nofollow but in sitemap
« Reply #1 on: July 08, 2010, 09:28:41 AM »
Hello,

what is an example URL that is included? Don't you have it specified in "Doi not parse" setting in sitemap generator configuration? (in this case it's still included)
Re: meta nofollow but in sitemap
« Reply #2 on: July 08, 2010, 05:01:25 PM »
hi!
an example would be [ External links are visible to forum administrators only ]
it has the meta tag in the source code: robots noindex, nofollow.

i automatically include this meta tag when a category page has 0 results. i don't want the searchengines to index these pages and so i don't want them to be in the sitemap too.

i can't add them manually to the exclude filter as this meta tag is only on pages with 0 results and they can differ from time to time as categories may have items in it in the future and others that have items now will not have them in the future maybe.

so the question is, does the generator look at the meta-tag, or ist the robots info on the page irrelevant. as i read, that robots.txt rules are followed by the generator i hoped that the page specific meta-tags are too ;)

thanks for your info about this!
mathias
Re: meta nofollow but in sitemap
« Reply #3 on: July 09, 2010, 08:18:01 AM »
when i look at this post https://www.xml-sitemaps.com/forum/index.php/topic,1868.html
it seems that the meta robots tag should also work for the generator
why is it not working for me?
m
Re: meta nofollow but in sitemap
« Reply #4 on: July 09, 2010, 09:59:06 AM »
Hello,

I just checked the URL you mentioned above and I don't see the robots meta tag in it's html source, please check that.
Re: meta nofollow but in sitemap
« Reply #5 on: July 09, 2010, 11:21:59 AM »
alright, try this one [ External links are visible to forum administrators only ] - i see the meta tag in the source code
maybe the other page was updatet in the meantime, i don't know any more which url it was as it his just visible for admins ;)
thanks for recheck!
Re: meta nofollow but in sitemap
« Reply #6 on: July 09, 2010, 12:53:28 PM »
Possibly the meta tag was not there at the moment when sitemap generator was crawloing your site? You can regenerate sitemap to check that.
Re: meta nofollow but in sitemap
« Reply #7 on: July 09, 2010, 03:34:19 PM »
i rechecked. the tag was there... what can the problem else be?
Re: meta nofollow but in sitemap
« Reply #9 on: July 13, 2010, 03:28:29 PM »
yeah, you already said that!
i can recheck and find what it is. i just want to knwo if it is a feature of the generator to automatically exclude pages with a robots noindex attribute. if it should work like this - i will find out as the problem seems to be on my side...
Re: meta nofollow but in sitemap
« Reply #10 on: July 14, 2010, 04:56:11 PM »
Yes, definitely - sitemap generator excludes pages with robots noindex meta tag (unless you have those pages specified in "do not parse" setting, as mentioend above)