XML Sitemaps Generator

Author Topic: Google have 11 000 URL restricted by robots  (Read 9943 times)

pinkstarfruit

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 40
Google have 11 000 URL restricted by robots
« on: May 29, 2011, 07:27:56 AM »
I have 11 000 url restricted by robots is that normal?


Google webmaster is blocking this kind of threads


/tags.php?tag=investment+in+thailand URL restricted by robots.txt

/threads/2207-Obama-Visit-To-Indonesia?p=2389 URL restricted by robots.txt

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Google have 11 000 URL restricted by robots
« Reply #1 on: May 29, 2011, 08:06:40 AM »
Hello,

if you don't want those pages to be indexed by search engines, you can keep the exclusion in robots.txt.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

pinkstarfruit

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 40
Re: Google have 11 000 URL restricted by robots
« Reply #2 on: May 29, 2011, 11:01:47 AM »
Yeah i know but should i?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Google have 11 000 URL restricted by robots
« Reply #3 on: May 30, 2011, 05:51:20 AM »
Perhaps you would want to index tags.php pages, as for the second example it looks like a duplicate URL for the topic.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2