Tools for Any SEO Challenge: Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous next »
Pages: 1

orders178

*
  • *
  • 4
exclude urls are being crawled
« on: October 20, 2017, 07:12:50 PM »
Hi there, I have urls like this in the exclude url area but they are being indexed.

wp-json/
?filter_by=
search.php?
mb3/

is there a limit to the amount of excludes you can have? I have a lot - could that be the issue?
Logged

XML-Sitemaps Support

*
  • *****
  • 11728
Re: exclude urls are being crawled
« Reply #1 on: October 21, 2017, 06:01:32 AM »
Hello,

that should be working.
What is an example URL that is added in sitemap and full list of exclusions?
Logged

orders178

*
  • *
  • 4
Re: exclude urls are being crawled
« Reply #2 on: October 24, 2017, 09:04:55 PM »
I am attaching a text file of the excludes and a sample of some of the urls that are getting into the sitemap that shouldn't be there. I will run it again to be sure that this is all the most accurate information for you.
Logged

XML-Sitemaps Support

*
  • *****
  • 11728
Re: exclude urls are being crawled
« Reply #3 on: October 25, 2017, 05:29:46 AM »
Hello,

if you are using "*" in Exclude URLs setting, then it must be defined in "regular expression" syntax:
https://en.wikipedia.org/wiki/Regular_expression

In your case, you need to escape "?" characters to avoid issues:
Code: [Select]
?filter_by=
search.php?should be replaced with:
Code: [Select]
\?filter_by=
search.php\?
Logged

orders178

*
  • *
  • 4
Re: exclude urls are being crawled
« Reply #4 on: October 25, 2017, 05:04:29 PM »
ok that makes a lot of sense! will give that a shot and let you know the results. thanks so much for your prompt replies
Logged

orders178

*
  • *
  • 4
Re: exclude urls are being crawled
« Reply #5 on: October 25, 2017, 06:08:05 PM »
That is working! thanks so much!
Logged

XML-Sitemaps Support

*
  • *****
  • 11728
Re: exclude urls are being crawled
« Reply #6 on: October 26, 2017, 04:42:41 AM »
Great, I'm glad it works!
Logged
Pages: 1
« previous next »