• Welcome to Sitemap Generator Forum.
 

How to detect xml-sitemap crawler traffic?

Started by joseluis1, March 17, 2022, 11:12:59 AM

Previous topic - Next topic

joseluis1

Hi, I am looking for a way to detect on our NGINX server which traffic is coming from the crawl.
I thought that there was a config to personalize the User-agent that the crawler used but have not found it or any doc regarding if it uses an specific one.

Is it using an specific one when crawling? Any other specific requested header I could use?
Thanks.

XML-Sitemaps Support

Hello,

xml-sitemaps user-agent http header is:
Mozilla/5.0 (compatible; XML Sitemaps Generator; [ External links are visible to logged in users only ]) Gecko XML-Sitemaps/1.0