Exclude Urls, Do not parse urls
« on: April 20, 2009, 05:50:16 AM »
Hello, I have a website www.domain.com
I  don't want to crawl and put links in sitemaps from directory www.domain.com/dir/
I write in configuration in field EXCLUDE URLS "www.domain.com/dir/" and in DO NOT PARSE field also "www.domain.com/dir/". But when the script start to crawl they crawl my directory DIR and put links to sitemap.
I don't want to put this dir to sitemap, how can I do that?

Thanks!
Re: Exclude Urls, Do not parse urls
« Reply #1 on: April 21, 2009, 01:31:50 AM »
You should define it without domain name, just:
Code: [Select]
dir/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Exclude Urls, Do not parse urls
« Reply #2 on: April 28, 2009, 09:10:56 PM »
I have Disallow: /*?quicktabs and Disallow: /*?sort= in the robot.txt and google webmaster tool shows as if blocked, but the sitemap generator includes the urls with these variables.

http://www.domain.com/taxonomy/term/nodetitle?quicktabs_1=0
http://www.domain.com/taxonomy/term/nodetitle?sort=desc&order=Topic&quicktabs_1=1

Would I just place the following in "Exclude URLs:" or how can I not include it into the sitemap.xml file?
/*?quicktabs
/*?sort=
Re: Exclude Urls, Do not parse urls
« Reply #3 on: April 29, 2009, 08:06:40 PM »
Hello,

in "Exclude URLs" option you should add them as:
Code: [Select]
?quicktabs
?sort=
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.