Can't exclude urls
« on: December 13, 2006, 05:05:56 AM »
I have added ext. to my Exclude URLs: box but they are still being crawled and added to my sitemap.html and urlstxt files.

I have tried different ways ie. blah.html, /blah.html , but still won't exclude.

also getting alot of urls like these in both of these files
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
Re: Can't exclude urls
« Reply #1 on: December 13, 2006, 10:49:11 PM »
Hello,

in order to exclude http://www.xxxxxxxxx.com/?this=XX URLs, you should add this in bot "Do not parse" and "Exclude URLs":

Code: [Select]
this=
Re: Can't exclude urls
« Reply #2 on: December 14, 2006, 01:26:33 AM »
Ok thanks for the help on excluding these type of links - [ External links are visible to forum administrators only ].

What about the first part of my question, excluding normal links that end in html?

Like this link - blah.html. I have put this in the exclude box in many different ways, ie:

blah.html
/blah.html
folder/blah.html

Anyway I put it it still ends being crawled and placed in the sitemap.
Re: Can't exclude urls
« Reply #3 on: December 14, 2006, 05:21:27 PM »
Ok I have added this link - [ External links are visible to forum administrators only ] - to the Do not Parse  and Exclude Urls bot and it is still being added to my sitemap. I have tried many configurations like;
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
/?this=1
?this=1
/?this=
?this=

They are not excluded

There are 46 of these numbered sequentially. I don't even know what these links are, they don't exist on my web site.
Re: Can't exclude urls
« Reply #4 on: December 14, 2006, 11:05:10 PM »
Quote
Like this link - blah.html. I have put this in the exclude box in many different ways, ie:

blah.html
Use this exclusion string:
Code: [Select]
.html
Quote
Ok I have added this link - http://www.xxxxxxxxx.com/?this=1 - to the Do not Parse  and Exclude Urls bot and it is still being added to my sitemap. I have tried many configurations like;
You should add the following for that:
Code: [Select]
this=1
Re: Can't exclude urls
« Reply #5 on: December 15, 2006, 06:48:03 PM »

Use this exclusion string:
Code: [Select]
.html
That isn't a solution. if I add that then all my links ending in .html are excluded. I only want to exclude certain links.

Quote
Ok I have added this link - [ External links are visible to forum administrators only ] - to the Do not Parse  and Exclude Urls bot and it is still being added to my sitemap. I have tried many configurations like;
You should add the following for that:
Code: [Select]
this=1[/quote]

This solution didn't work either. What are these links? ( this=1, this=2. etc ). Like I said they don't exist in my website and are being generated by the Sitemap Generator.
Re: Can't exclude urls
« Reply #6 on: December 15, 2006, 07:36:04 PM »
Quote
This solution didn't work either. What are these links? ( this=1, this=2. etc ). Like I said they don't exist in my website and are being generated by the Sitemap Generator.
Sitemap Generator doesn't create ANY links at your website, it only finds existing ones.