Toggle navigation
PRO Sitemaps
Standalone Generator
About Sitemaps
News
SEO Tools
Sitemap Generator Forum
Guest
Login
Register
Menu
Home
Help
Search
Login
Register
XML Sitemaps Discussions Category
/
PRO Sitemap Generator Service
/
Robots.txt empty
On Page SEO Checker:
Get a list of improvements for your website for free in less than 5 minutes!
« previous
next »
Pages:
1
frankic
6
Robots.txt empty
«
on:
April 13, 2012, 01:10:04 AM »
Should the robots.txt be empty? How can we manually add information to robots.txt? I do not see an option for adding code anywhere for the robots file. I know you can add or disallow URLs but where can we actually add info to robot file?
Logged
XML-Sitemaps Support
11767
Re: Robots.txt empty
«
Reply #1 on:
April 13, 2012, 02:19:07 PM »
Hello,
empty robots.txt allows all pages to be indexed (which is ok).
If you want to exclude a folder, for instance, you can add this:
User-agent: *
Disallow: /foldername/
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
frankic
6
Re: Robots.txt empty
«
Reply #2 on:
April 13, 2012, 04:42:18 PM »
where do you add these lines?
Logged
XML-Sitemaps Support
11767
Re: Robots.txt empty
«
Reply #3 on:
April 16, 2012, 09:55:52 PM »
In robots.txt file (create a new file if you don't have one yet)
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
Pages:
1
« previous
next »