• Welcome to Sitemap Generator Forum.
 

Robots.txt can not be passed

Started by office60, October 22, 2011, 04:36:59 PM

Previous topic - Next topic

office60

Hello,

In my robots.txt I have some folders that I do not want to be indexed by Google and by the sitemap.

The problem is that the generator can not pass the robots.txt file and so, even if my site has over 2.000 pages, the generator indexes only 35.

Is there a way to make the generator crawl all my pages except the ones mentioned in the robots.txt?


Thanks.