User Agent for Crawling

[RobertMidd]

Hi,

I purchased the Unlimited PHP Sitemap Generator today and have now installed and tried to create a sitemap.

My site recognises the user-agent as a BOT and then gives friendly url's for SEO.

I need to be able to crawl with a user-agent that is a recognised BOT and thought by changing the user-agent in firefox to Googlebot/2.1 (+[ External links are visible to forum administrators only ]) that this would work but it does not work on the crawl but works ok when I goto my site.

How can I force the crawl with a user-agent for a BOT ?

[XML-Sitemaps Support]

Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',

[tommixoft2]

Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',

Hi i can't find it in V3.0, where it can be?

[XML-Sitemaps Support]

Hello,

in v3.0 you should define it in generator/data/generator.conf file, add the following line:

Code Select Expand

<option name="xs_crawl_ident">your custom string</option>