Robots.txt can not be passed
« on: October 22, 2011, 04:36:59 PM »
Hello,

In my robots.txt I have some folders that I do not want to be indexed by Google and by the sitemap.

The problem is that the generator can not pass the robots.txt file and so, even if my site has over 2.000 pages, the generator indexes only 35.

Is there a way to make the generator crawl all my pages except the ones mentioned in the robots.txt?


Thanks.
Re: Robots.txt can not be passed
« Reply #1 on: October 22, 2011, 08:27:36 PM »
Hello,

replied to your email.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.