XML Sitemaps Generator

Author Topic: User-agent in robots.txt  (Read 22159 times)

wouter

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
User-agent in robots.txt
« on: February 02, 2007, 11:02:19 AM »
Hi,

I want to dissalow my forum directory for the sitemap. Because my forum is to big. So i have set in the robots.txt file the follow.

User-agent: XML Sitemaps Generator 1.0 (http://www.xml-sitemaps.com/)
Disallow: forum/

And have in de config 'xs_robotstxt' => '1',

But he will index the forum directory so i think the user-agent issnt right. Wat most i set ?

Greets

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: User-agent in robots.txt
« Reply #1 on: February 02, 2007, 10:32:57 PM »
Hello,

sitemap generator is looking for "googlebot" (or "*") user-agent in robots.txt, since it's main purpose is to create google sitemap. You can add "forum/" in "Do not parse" and "Exclude URLs" options to achieve the result you need.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

wouter

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: User-agent in robots.txt
« Reply #2 on: February 09, 2007, 03:13:56 PM »
[external links are visible to admins only]
Hello,

sitemap generator is looking for "googlebot" (or "*") user-agent in robots.txt, since it's main purpose is to create google sitemap. You can add "forum/" in "Do not parse" and "Exclude URLs" options to achieve the result you need.

I have to do that in robots.txt ?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: User-agent in robots.txt
« Reply #3 on: February 09, 2007, 11:44:38 PM »
"Do not parse" and "Exclude URLs" are defined in sitemap generator configuration (not robots.txt).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2