User-agent in robots.txt

[wouter]

Hi,

I want to dissalow my forum directory for the sitemap. Because my forum is to big. So i have set in the robots.txt file the follow.

User-agent: XML Sitemaps Generator 1.0 (http://www.xml-sitemaps.com/)
Disallow: forum/

And have in de config 'xs_robotstxt' => '1',

But he will index the forum directory so i think the user-agent issnt right. Wat most i set ?

Greets

[XML-Sitemaps Support]

Hello,

sitemap generator is looking for "googlebot" (or "*") user-agent in robots.txt, since it's main purpose is to create google sitemap. You can add "forum/" in "Do not parse" and "Exclude URLs" options to achieve the result you need.

[wouter]

Hello,

sitemap generator is looking for "googlebot" (or "*") user-agent in robots.txt, since it's main purpose is to create google sitemap. You can add "forum/" in "Do not parse" and "Exclude URLs" options to achieve the result you need.

I have to do that in robots.txt ?

[XML-Sitemaps Support]

"Do not parse" and "Exclude URLs" are defined in sitemap generator configuration (not robots.txt).