User-agent

Introduction

A user agent is any software that “retrieves, renders, and facilitates end-user interaction with Web content” on behalf of a user. As a result, a user agent is a subset of a software agent. Web browsers and email clients are two well-known examples of user agents. In a client–server system, a user agent frequently serves as the client.

In some cases, such as the Session Initiation Protocol (SIP), the term user agent refers to both ends of a communication session.

When a software agent operates in a network protocol, it frequently identifies itself, its application type, operating system, device model, software vendor, or software revision to its operating peer by submitting a characteristic identification string.

Use within the HTTP protocol

Source: Mozilla Developer Wiki under CC-BY-SA 2.5

The User-Agent string is frequently used in HTTP for content negotiation, in which the origin server selects appropriate content or operating parameters for the response.

A web server, for example, may use the User-Agent string to select variants based on the known capabilities of a specific version of client software. RFC 1945 incorporates the concept of content tailoring into the HTTP standard “for the purpose of tailoring responses to avoid particular user agent limitations.”

Standard for user operated web browsers

Section 5.5.3 of HTTP/1.1 standards (Semantics and Content) currently specifies the User-Agent string format. In HTTP, the User-Agent string takes the form of a list of product tokens (keywords) with optional comments.

If a user’s product is called WikiBrowser, their user agent string could be WikiBrowser/1.0 Gecko/1.0. The “most important” component of the product is listed first.

This string’s components are as follows:

The product name and version (WikiBrowser/1.0)
The layout engine used and it’s version (Gecko/1.0)

Some websites and web-based applications tailor their experience and serve content in a different way based on the user-agent of the incoming session. For this reason many modern browsers such as Chrome, Firefox and Opera include multiple strings in their user-agents in order to get the best compatibility with different pages online.

For example, Safari on iOS devices uses the following string for it’s user-agent:

Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405

(Example of a Safari user-agent string on an Apple iPad)

Common user-agents

Mozilla Firefox (Gecko)

Mozilla’s Firefox browser is known to use a variety of user-agents depending on it’s version. Mozilla created the Gecko engine which is used in older versions of the browser.

Here is an example string of Firefox version 77 on a Windows PC:

• Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0
• Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0

Safari (Apple Webkit)

Similar to Mozilla Apple created their own rendering engine called Webkit which is used by other browsers as well. Here is an example of a string from Safari version 7.0.3 on an Apple Macintosh computer:

• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A

Bingbot (automated crawler)

Bingbot is a web-crawling robot (a type of internet bot) that Microsoft deployed in October 2010 to supply Bing with information about online content for search results.

It gathers web documents to create a searchable index for Bing (search engine). It serves the same purpose as Google’s Googlebot.

“Mozilla/5.0 (compatible; bingbot/2.0; + http://www.bing.com/bingbot.htm)” is a common user agent string for Bingbot. This appears in the web server logs to notify the webmaster of a file request.

Googlebot (automated crawler)

Generally, Googlebot refers to Google’s web crawler, which consists of two types of software: a desktop crawler that mimics a user on desktop and a mobile crawler that mimics a user on a mobile device.

The Googlebot is used by Google to scour the web and collect information about websites and content so they can be indexed and included inside search results.

Google uses several different user-agents for crawling different types of content such as:

• Googlebot-Image: Used to collect and index image related content
  
• Googlebot-News: Used to gather information for Google news
• Googlebot: General crawler used to gather standard web pages

Mozilla Thunderbird (email client)

Mozilla Thunderbird is a popular opensource email reading application. When used it sends it’s user-agent across network traffic requests which include the following strings:

• Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Lightning/4.0.2
• Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Lightning/4.0.2
• Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Lightning/4.0.2
• Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0
• Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Lightning/4.0.2
• Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Lightning/4.0.2

The type of Operating System used such as Windows, macOS or Linux and the version used along with the version of Thunderbird itself will cause the string to differ slightly so these examples may be out-of-date but the same structure is used with newer information for version numbers.

User-agent spoofing

The popularity of various Web browser products has fluctuated throughout the history of the Web, and this has influenced website design. Websites frequently include code to detect browser version in order to modify the page design sent based on the user agent string received.

This may imply that less-popular browsers are not sent complex content (even if they are capable of handling it) or, in extreme cases, refuse all content.

As a result technology has been developed to display a fake user-agent or “spoof” the user-agent shown to a server by a browser.

This can cause the server to serve content which otherwise wouldn’t be served  to the browser. An example of this would be a website that blocks desktop computers from viewing certain video. Utilizing a mobile user-agent impersonating a browser for iOS or Android would play the video even though desktop browsers may have been blocked.

Spoofing the user-agent can be used legitimately in order to access content intended for other browsers however automated spambots or programs can take advantage of this and pretend to be legitimate user-agents or browsers like Firefox or GoogleBot.

Eventual depreciation by some browsers

Google announced that it would eliminate support for the User-Agent header in its Google Chrome web browser. Other browser makers such as Apple, Microsoft and Mozilla were in support of the move, but it isn’t known at the time of publishing when or if other browser vendors will follow.

Conclusion

We hope you found this page to be helpful and provide a detailed explanation of user-agents. Be sure to check out our blog and follow us on social media for more content.

Also check out our business services such as our Australian based SSD eCommerce Web hosting and other business solutions.

Related content from our glossary:

World Wide Web (WWW)

Cloud Computing

API (Application Programming Interface)

Source(s) cited:

“User agent” Wikipedia, 25 Feb. 2004, en.wikipedia.org/wiki/User_agent. Accessed 28 Jan. 2022.

“What Is Googlebot | Google Search Central | Google Developers” Google Search Central | Google Developers, developers.google.com/search/docs/advanced/crawling/googlebot. Accessed 29 Jan. 2022. under https://creativecommons.org/licenses/by/4.0/

“Bingbot” Wikipedia, 25 Dec. 2010, en.wikipedia.org/wiki/Bingbot. Accessed 29 Jan. 2022.

“Google Crawler (User Agent) Overview | Google Search Central | Google Developers” Google Search Central | Google Developers, developers.google.com/search/docs/advanced/crawling/overview-google-crawlers. Accessed 29 Jan. 2022.