1. History

2. Description
     U+0000 to U+D7FF and U+E000 to U+FFFF    U+010000 to U+10FFFF    U+D800 to U+DFFF    Examples  

3. {{anchor|UTF-16LE|UTF-16BE}}Byte order encoding schemes

4. Usage

5. See also

6. Notes

7. References

8. External links

UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode. The encoding is variable-length, as code points are encoded with one or two 16-bit code units (also see comparison of Unicode encodings for a comparison of UTF-8, -16 & -32).

UTF-16 arose from an earlier fixed-width 16-bit encoding known as UCS-2 (for 2-byte Universal Character Set) once it became clear that more than 2¹⁶ code points were needed.^[1]

UTF-16 is used internally by systems such as Windows and Java and by JavaScript, and often for plain text and for word-processing data files on Windows. It is rarely used for files on Unix/Linux or macOS. It never gained popularity on the web, where UTF-8 is dominant (and considered "the mandatory encoding for all [text]" by WHATWG^[2]). UTF-16 is used by under 0.01% of web pages themselves.^[2] WHATWG recommends that for security reasons browser apps should not use UTF-16.^[3]

History

In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" (UCS) that would replace earlier language-specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (2¹⁶) values, which would require 2 bytes per character. Two groups worked on this in parallel, the IEEE and the Unicode Consortium, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was usually called "Unicode", but is now called "UCS-2". UCS-2 differs from UTF-16 by being a constant length encoding^[4] and only capable of encoding characters of BMP. It is supported by many programs.

Early in this process it became increasingly clear that 2¹⁶ characters would not suffice,^[1] and IEEE introduced a larger 31-bit space and an encoding (UCS-4) that would require 4 bytes per character. This was resisted by the Unicode Consortium, both because 4 bytes per character wasted a lot of disk space and memory, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise to resolve this impasse in version 2.0 of the Unicode standard in July 1996^[5] and is fully specified in RFC 2781 published in 2000 by the IETF.^[6][7]

In UTF-16, code points greater or equal to 2¹⁶ are encoded using two 16-bit code units. The standards organizations chose the largest block available of un-allocated 16-bit code points to use as these code units. Unlike UTF-8 they did not provide a means to encode these code points.

UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard."^[8] There are no plans to extend UTF-16 to support a higher number of code points, or the codes replaced by surrogates, as allocating code points for this would violate the Unicode Stability Policy with respect to general category or surrogate code points.^[9] An example idea would be to allocate another BMP value to prefix a triple of low,low,high surrogates (the order swapped so that it cannot match a surrogate pair in searches), allowing 2³⁰ more code points to be encoded, but changing the purpose of a code point is disallowed (using no prefix is also not allowed as two of these characters next to each other would match a surrogate pair).

Description

U+0000 to U+D7FF and U+E000 to U+FFFF

Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the Basic Multilingual Plane (BMP) are the only code points that can be represented in UCS-2.{{Citation needed|date=November 2015}} As of Unicode 9.0 some modern non-Latin Asian, Middle-Eastern and African scripts fall outside this range, as do most emoji characters.

U+010000 to U+10FFFF

Code points from the other planes (called Supplementary Planes) are encoded as two 16-bit code units called a surrogate pair, by the following scheme:

 !{{diagonal split header|High | Low}} !DC00 !DC01 !   …    !DFFF

• 0x10000 is subtracted from the code point, leaving a 20-bit number in the range 0x00000–0xFFFFF.
• The high ten bits (in the range 0x000–0x3FF) are added to 0xD800 to give the first 16-bit code unit or high surrogate, which will be in the range {{color|#920000|0xD800–0xDBFF}}.
• The low ten bits (also in the range 0x000–0x3FF) are added to 0xDC00 to give the second 16-bit code unit or low surrogate, which will be in the range {{color|#000092|0xDC00–0xDFFF}}.

The high surrogate and low surrogate are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8.^[10]

Since the ranges for the high surrogates ({{color|#920000|0xD800–0xDBFF}}), low surrogates ({{color|#000092|0xDC00–0xDFFF}}), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are disjoint, it is not possible for a surrogate to match a BMP character, or for two adjacent code units to look like a legal surrogate pair. This simplifies searches a great deal. It also means that UTF-16 is self-synchronizing on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units (i.e. the type of code unit can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as Shift JIS and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string (UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte).

Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. {{CVE|2008-2938|2012-2135}}).

The Supplementary Planes contain emoji, historic scripts, less used symbols, less used Chinese ideographs, etc. Since the encoding of Supplementary Planes contains 20 significant bits (10 of 16 bits in each of the high and low surrogates), 2²⁰ code points can be encoded, divided into 16 planes of 2¹⁶ code points each. Including the separately-handled Basic Multilingual Plane, there are a total of 17 planes.

U+D800 to U+DFFF

The Unicode standard permanently reserves these code point values for UTF-16 encoding of the high and low surrogates, and they will never be assigned a character, so there should be no reason to encode them. The official Unicode standard says that no UTF forms, including UTF-16, can encode these code points.

However UCS-2, UTF-8, and UTF-32 can encode these code points in trivial and obvious ways, and large amounts of software does so even though the standard states that such arrangements should be treated as encoding errors.

It is possible to unambiguously encode an unpaired surrogate (a high surrogate code point not followed by a low one, or a low one not proceeded by a high one) in UTF-16 by using a code unit equal to the code point. The majority of UTF-16 encoder and decoder implementations do this then when translating between encodings.{{citation needed|date=October 2011}} Windows allows unpaired surrogates in filenames and other places, which generally means they have to be supported by software in spite of their exclusion from the Unicode standard.

Examples

To encode U+10437 (𐐷) to UTF-16:

• Subtract 0x10000 from the code point, leaving 0x0437.
• For the high surrogate, shift right by 10 (divide by 0x400), then add 0xD800, resulting in 0x0001 + 0xD800 = 0xD801.
• For the low surrogate, take the low 10 bits (remainder of dividing by 0x400), then add 0xDC00, resulting in 0x0037 + 0xDC00 = 0xDC37.

To decode U+10437 (𐐷) from UTF-16:

• Take the high surrogate (0xD801) and subtract 0xD800, then multiply by 0x400, resulting in 0x0001 × 0x400 = 0x0400.
• Take the low surrogate (0xDC37) and subtract 0xDC00, resulting in 0x37.
• Add these two results together (0x0437), and finally add 0x10000 to get the final decoded UTF-32 code point, 0x10437.

The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.

{{anchor|UTF-16LE|UTF-16BE}}Byte order encoding schemes

UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the endianness (byte order) of the computer architecture.

To assist in recognizing the byte order of code units, UTF-16 allows a Byte Order Mark (BOM), a code point with the value U+FEFF, to precede the first actual coded value.^[11] (U+FEFF is the invisible zero-width non-breaking space/ZWNBSP character.)^[12] If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the non-character value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values.

If the BOM is missing, RFC 2781 says that big-endian encoding should be assumed. In practice, due to Windows using little-endian order by default, many applications similarly assume little-endian encoding by default. It is also reliable to detect endianess by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it is big-endian.

The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically not supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule.

For Internet protocols, IANA has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings (the names are case insensitive). The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.

Similar designations, UCS-2BE and UCS-2LE, are used to show versions of UCS-2.

Usage

UTF-16 is used for text in the OS API of all currently supported versions of Microsoft Windows (and including at least all since Windows CE/2000/XP/2003/Vista/7^[13]) including Windows 10 (while since insider build 17035 and the April 2018 update, it has improved UTF-8 support in addition to UTF-16; see Unicode in Microsoft Windows#UTF-8). Older Windows NT systems (prior to Windows 2000) only support UCS-2.^[14] In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages.^[15][16] Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings.

UTF-16 is used by the Qualcomm BREW operating systems; the .NET environments; and the Qt cross-platform graphical widget toolkit.

The Joliet file system, used in CD-ROM media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name).

The Python language environment officially only uses UCS-2 internally since version 2.0, but the UTF-8 decoder to "Unicode" produces correct UTF-16. Since Python 2.2, "wide" builds of Unicode are supported which use UTF-32 instead;^[19] these are primarily used on Linux. Python 3.3 no longer ever uses UTF-16, instead an encoding that gives the most compact representation for the given string is chosen from ASCII/Latin-1, UCS-2, and UTF-32.^[20]

Java originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0.

JavaScript implementations may use UCS-2 or UTF-16.^[21] As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective.

In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style "\\uXXXX" syntax explicitly limits itself to 4 hex digits. The most common (used by C++, C#, D, and several other languages) is to use an upper-case 'U' with 8 hex digits such as "\\U0001D11E".^[22] In Java 7 regular expressions, ICU, and Perl, the syntax "\\x{1D11E}" must be used; similarly, in ECMAScript 2015 (JavaScript), the escape format is "\\u{1D11E}". In many other cases (such as Java outside of regular expressions),^[23] the only way to get non-BMP characters is to enter the surrogate halves individually, for example: "\\uD834\\uDD1E" for U+1D11E.

String implementations based on UTF-16 typically return lengths and allow indexing in terms of code units, not code points. Neither code points nor code units correspond to anything an end user might recognise as a “character”; the things users identify as characters may in general consist of a base code point and a sequence of combining characters (or be a sequence of code points of other kind, for example Hangul conjoining jamos){{snd}} Unicode refers to this as a grapheme cluster^[24]{{snd}} and as such, applications dealing with Unicode strings, whatever the encoding, have to cope with the fact that they cannot arbitrarily split and combine strings.

UCS-2 is supported by PHP^[25] and MySQL.^[4]

See also

• Comparison of Unicode encodings
• Unicode plane
• UTF-8
• UTF-32

Notes

References

External links

• [https://www.unicode.org/faq/utf_bom.html#utf16-4 A very short algorithm for determining the surrogate pair for any code point]
• [https://www.unicode.org/notes/tn12/ Unicode Technical Note #12: UTF-16 for Processing]
• [https://www.unicode.org/faq/utf_bom.html#utf16-11 Unicode FAQ: What is the difference between UCS-2 and UTF-16?]
• [https://www.unicode.org/charts/charindex.html Unicode Character Name Index]
• RFC 2781: UTF-16, an encoding of ISO 10646
• [https://docs.oracle.com/javase/6/docs/api/java/lang/String.html#charAt(int) java.lang.String documentation, discussing surrogate handling]

• Turhan Bey
• Turhan Pasha Permeti
• Turhan Përmeti
• Turia
• Turiacus
• Turian
• Turia (river)
• Turicum (automobile)
• Turiec (Sajó)
• Turiec (Váh)
• Turin Brakes
• Turin Cathedral
• Turin (disambiguation)
• Turin, GA
• Turing, A.M.
• Turing Award
• Turing award
• Turing Award Laureate
• Turing Award laureates
• Turing (cipher)
• Turing-complete
• Turing Complete
• Turing-Complete
• Turing-completeness
• Turing completeness