XML Sitemaps Generator

Author Topic: User Agent for Crawling  (Read 17336 times)

RobertMidd

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
User Agent for Crawling
« on: May 21, 2008, 04:25:07 PM »
Hi,

I purchased the Unlimited PHP Sitemap Generator today and have now installed and tried to create a sitemap.

My site recognises the user-agent as a BOT and then gives friendly url's for SEO.

I need to be able to crawl with a user-agent that is a recognised BOT and thought by changing the user-agent in firefox to Googlebot/2.1 (+http://www.googlebot.com/bot.html) that this would work but it does not work on the crawl but works ok when I goto my site.

How can I force the crawl with a user-agent for a BOT ?
« Last Edit: May 21, 2008, 04:37:22 PM by RobertMidd »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: User Agent for Crawling
« Reply #1 on: May 21, 2008, 09:16:28 PM »
Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

tommixoft2

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: User Agent for Crawling
« Reply #2 on: July 29, 2009, 01:37:23 PM »
[external links are visible to admins only]
Hello,

you can define the following in config.inc.php file:
    'xs_crawl_ident' => 'your custom user-agent string',


Hi i can't find it in V3.0, where it can be?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: User Agent for Crawling
« Reply #3 on: July 29, 2009, 03:39:52 PM »
Hello,

in v3.0 you should define it in generator/data/generator.conf file, add the following line:
Code: [Select]
<option name="xs_crawl_ident">your custom string</option>
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2