User Agent for Crawling
« on: May 21, 2008, 05:25:07 PM »
Hi,

I purchased the Unlimited PHP Sitemap Generator today and have now installed and tried to create a sitemap.

My site recognises the user-agent as a BOT and then gives friendly url's for SEO.

I need to be able to crawl with a user-agent that is a recognised BOT and thought by changing the user-agent in firefox to Googlebot/2.1 (+http://www.googlebot.com/bot.html) that this would work but it does not work on the crawl but works ok when I goto my site.

How can I force the crawl with a user-agent for a BOT ?
« Last Edit: May 21, 2008, 05:37:22 PM by RobertMidd »
Re: User Agent for Crawling
« Reply #1 on: May 21, 2008, 10:16:28 PM »
Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',
Re: User Agent for Crawling
« Reply #2 on: July 29, 2009, 02:37:23 PM »
Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',


Hi i can't find it in V3.0, where it can be?
Re: User Agent for Crawling
« Reply #3 on: July 29, 2009, 04:39:52 PM »
Hello,

in v3.0 you should define it in generator/data/generator.conf file, add the following line:
Code: [Select]
<option name="xs_crawl_ident">your custom string</option>