Toggle navigation
PRO Sitemaps
Standalone Generator
About Sitemaps
News
SEO Tools
Sitemap Generator Forum
Guest
Login
Register
Menu
Home
Help
Search
Login
Register
XML Sitemaps Discussions Category
/
Bug reporting for Standalone Sitemap Generator
/
exclude urls are being crawled
Keyword suggestion tool:
Pick the right keywords to target in your SEO and PPC campaigns
« previous
next »
Pages:
1
orders178
4
exclude urls are being crawled
«
on:
October 20, 2017, 07:12:50 PM »
Hi there, I have urls like this in the exclude url area but they are being indexed.
wp-json/
?filter_by=
search.php?
mb3/
is there a limit to the amount of excludes you can have? I have a lot - could that be the issue?
Logged
XML-Sitemaps Support
11706
Re: exclude urls are being crawled
«
Reply #1 on:
October 21, 2017, 06:01:32 AM »
Hello,
that should be working.
What is an example URL that is added in sitemap and full list of exclusions?
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
orders178
4
Re: exclude urls are being crawled
«
Reply #2 on:
October 24, 2017, 09:04:55 PM »
I am attaching a text file of the excludes and a sample of some of the urls that are getting into the sitemap that shouldn't be there. I will run it again to be sure that this is all the most accurate information for you.
Logged
XML-Sitemaps Support
11706
Re: exclude urls are being crawled
«
Reply #3 on:
October 25, 2017, 05:29:46 AM »
Hello,
if you are using "*" in Exclude URLs setting, then it must be defined in "regular expression" syntax:
https://en.wikipedia.org/wiki/Regular_expression
In your case, you need to escape "?" characters to avoid issues:
Code:
[Select]
?filter_by=
search.php?
should be replaced with:
Code:
[Select]
\?filter_by=
search.php\?
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
orders178
4
Re: exclude urls are being crawled
«
Reply #4 on:
October 25, 2017, 05:04:29 PM »
ok that makes a lot of sense! will give that a shot and let you know the results. thanks so much for your prompt replies
Logged
orders178
4
Re: exclude urls are being crawled
«
Reply #5 on:
October 25, 2017, 06:08:05 PM »
That is working! thanks so much!
Logged
XML-Sitemaps Support
11706
Re: exclude urls are being crawled
«
Reply #6 on:
October 26, 2017, 04:42:41 AM »
Great, I'm glad it works!
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
Pages:
1
« previous
next »