• Welcome to Sitemap Generator Forum.
 

Robots.txt empty

Started by frankic, April 13, 2012, 01:10:04 AM

Previous topic - Next topic

frankic

Should the robots.txt be empty? How can we manually add information to robots.txt? I do not see an option for adding code anywhere for the robots file. I know you can add or disallow URLs but where can we actually add info to robot file?

XML-Sitemaps Support

Hello,

empty robots.txt allows all pages to be indexed (which is ok).
If you want to exclude a folder, for instance, you can add this:
User-agent: *
Disallow: /foldername/

frankic

where do you add these lines?