XML Sitemaps Generator

Author Topic: Robots.txt empty  (Read 10004 times)

frankic

  • Approved member
  • *
  • Posts: 6
Robots.txt empty
« on: April 13, 2012, 12:10:04 AM »
Should the robots.txt be empty? How can we manually add information to robots.txt? I do not see an option for adding code anywhere for the robots file. I know you can add or disallow URLs but where can we actually add info to robot file?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Robots.txt empty
« Reply #1 on: April 13, 2012, 01:19:07 PM »
Hello,

empty robots.txt allows all pages to be indexed (which is ok).
If you want to exclude a folder, for instance, you can add this:
User-agent: *
Disallow: /foldername/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

frankic

  • Approved member
  • *
  • Posts: 6
Re: Robots.txt empty
« Reply #2 on: April 13, 2012, 03:42:18 PM »
where do you add these lines?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Robots.txt empty
« Reply #3 on: April 16, 2012, 08:55:52 PM »
In robots.txt file (create a new file if you don't have one yet)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2