Excluding Page & Sort URLs
« on: May 06, 2011, 10:28:58 PM »
Hi,

I understand how to exclude folders when they come right after the main domain in the URL. For example, if I want to exclude http://stageagent.com/privatecontent I know to exclude /privatecontent. However, how do I exclude crawling content (such as pagination & sorts) that do not easily fall into that category?

For example, I DO want to crawl http://stageagent.com/Auditions/Loc/Atlanta. But I don't want to crawl http://stageagent.com/Auditions/loc/Atlanta/sort/title or
http://stageagent.com/Auditions/loc/Atlanta/sort/created_datetime.

How can I exclude URLs that contain /sort/title or sort/created_datetime.

Another example: I do want to crawl http://stageagent.com/Shows/View/846.
But i don't want to crawl http://stageagent.com/Shows/View/846/name_ASC/#characters
or
http://stageagent.com/Shows/View/846/gender_ASC/#characters

How can I exclude URLs that contain name_ASC or gender_ASC?

Thank you,
Jason
Re: Excluding Page & Sort URLs
« Reply #1 on: May 07, 2011, 08:24:58 AM »
Hello,

it can be achieved by adding this in "Exclude URLs" setting:
Code: [Select]
sort/
_ASC
that will remove all links that have that included as a substring.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.