XML Sitemaps Generator

Author Topic: Robots.txt can not be passed  (Read 7692 times)

office60

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Robots.txt can not be passed
« on: October 22, 2011, 03:36:59 PM »
Hello,

In my robots.txt I have some folders that I do not want to be indexed by Google and by the sitemap.

The problem is that the generator can not pass the robots.txt file and so, even if my site has over 2.000 pages, the generator indexes only 35.

Is there a way to make the generator crawl all my pages except the ones mentioned in the robots.txt?


Thanks.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Robots.txt can not be passed
« Reply #1 on: October 22, 2011, 07:27:36 PM »
Hello,

replied to your email.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2