Free images tag: User Agents

Why does the Google-safety bot appear to visit your site?

Is the Google-Safety bot considered to be a good or bad bot and what does it mean when the safety bot visits?

User Agents

User Agents (UA) can be easily spoofed, so just because the logs show a visit from Google-safety doesn't mean that it is guaranteed to be from Google. You must check the IP that the bot visits from is a google bot IP address. Simple enough to check, but this can be easily overlooked.

Google-Safety

The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. This user agent ignores robots.txt rules.

Two key things mentioned by Google are:

• The safety bot ignores robots.txt
• Malware discovery

ignoring the robots.txt is a big thing, you could expect to see the safety bot in places you have excluded the other Google bots from visiting. Malware can't hide behind robots.txt, which is a good thing.

Malware discovery sounds like a good thing to do, that way the SERPS (Search Engine Results Page) can warn a user about malware on a site before they visit (or choose not to visit) the link.

Another Reason

There are other, undocumented reasons that Google-safety bot can visit your site and one of those we have seen was when a link is added to a YouTube video description. The bot will check the link in the video description, presumably for the malware part of the bot's job.

A way that Google can clamp down on malware links spreading through YouTube while ignoring robots.txt directives.

So don't be alarmed by the Google-Safety bot visiting. It doesn't necessarily mean that the big G considers your site to be unsafe or to be a source of malware. It may be good that YouTube uploaders recognise your site and Google comes to visit (even if it is a malware-sniffing, robots-ducking safety bot).

You may see some very old browsers or misconfigured UserAgents for browsers in your logs. These odd user agents are seen visiting from IP addresses and ASNs linked to Hosting companies.

You can expect these to be some bot, a computer script that visited websites for whatever reason the person that has decided to do that for. These bots that try to hide genuine human visitors provide no benefit but do take up resources that could be used to show pages to genuine visitors.

User-Agent Header

The User-Agent header can be set to almost anything you like; it can even be 'blank' and not contain any information. The User-Agent is helpful but not to be relied upon for genuine information. The helpfulness comes with the information that describes the type of rendering engine and capabilities the browser has. This means the website could tailor the page to match the capabilities of the browser. Less useful now because the web has moved forward, but back in the early days of the World Wide Web, the UA - User-Agent header in the browser was a helpful hint. There are extensions for Chrome and Firefox that make altering the User-Agent a simple task.

Why?

Why use an old, misconfigured or incorrect User agent to identify a bot? The only reason for a computer program to be configured to masquerade as a genuine human is to hide in plain sight. To slip through security filters to trick web admins into thinking these user agents are linked to real human visitors. Why do they make spelling or configuration mistakes? I don't know, but it could be just laziness or a way to leave their mark.

Good bots

Good bots can be helpful, like the well-known Google Bots that crawl pages to index content for their search engine. The Google bots clearly identify themselves, so it is easy to block them and count the visits. This is a good example of a good bot, and no deception is employed in the visits.

Lists

Here are a few of those User Agents seen in the wild that look like human visitors with old or unusual browsers but come from Hosting addresses linked to spam, scraping and probing.

• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.1.25 (KHTML, like Gecko) Version/8.0 Safari/600.1.25
• Mozilla/6.4 (Windows NT 11.1) Gecko/2010102 Firefox/99.0
• Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 (2b7146e2-bc0c-4240-ae78-608c0c43c275)
• Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.3103.400 QQBrowser/9.6.11372.400
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Edge/76.0.1655.82 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19577
• Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.772.39 Safari/537.36
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.2925.105 Safari/537.36
• Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 OPR/45.0.2552.898
• Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0
• Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
• Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
• Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.76 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2656.18 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4; rv:82.0) Gecko/20100101 Firefox/82.0
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5; rv:76.0) Gecko/20100101 Firefox/76.0
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.74 Safari/537.36 Edg/79.0.309.43
• Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0
• Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
• Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 YaBrowser/23.5.3.904 Yowser/2.5 Safari/537.36
• Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36
• Mozilla/5.0 (X11; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0
• Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0
• Opera/9.80 (Macintosh; Intel Mac OS X 10_13_6; U; en) Presto/2.2.15 Version/10.10