How to detect xml-sitemap crawler traffic?
« on: March 17, 2022, 11:12:59 AM »
Hi, I am looking for a way to detect on our NGINX server which traffic is coming from the crawl.
I thought that there was a config to personalize the User-agent that the crawler used but have not found it or any doc regarding if it uses an specific one.

Is it using an specific one when crawling? Any other specific requested header I could use?
Thanks.
Re: How to detect xml-sitemap crawler traffic?
« Reply #1 on: March 17, 2022, 11:17:49 AM »
Hello,

xml-sitemaps user-agent http header is:
Mozilla/5.0 (compatible; XML Sitemaps Generator; www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0