Robots.txt empty
« on: April 13, 2012, 01:10:04 AM »
Should the robots.txt be empty? How can we manually add information to robots.txt? I do not see an option for adding code anywhere for the robots file. I know you can add or disallow URLs but where can we actually add info to robot file?
Re: Robots.txt empty
« Reply #1 on: April 13, 2012, 02:19:07 PM »
Hello,

empty robots.txt allows all pages to be indexed (which is ok).
If you want to exclude a folder, for instance, you can add this:
User-agent: *
Disallow: /foldername/
Re: Robots.txt empty
« Reply #2 on: April 13, 2012, 04:42:18 PM »
where do you add these lines?