What is UTF-8: definition, operation, errors and uses

Informatec Digital » Computing » What is UTF-8: definition, operation, errors and uses

If you're reading this article today without seeing any strange symbols, it's thanks to UTF-8 . This encoding allows letters, accents, technical symbols, and even emojis to be displayed the same way in any modern browser, operating system, or email client. It's the most widespread standard on the web and the foundation of digital communication as we know it.

When a device displays text, it's actually processing numbers . These numbers are code points defined by the Unicode standard , and to convert them into bytes that travel over a network or are saved to a file, we perform a transformation: UTF-8 . In the following lines, you'll understand what it is, how it works, why it became standard, its advantages and limitations, and how to avoid common errors.

What is UTF-8?

UTF-8 (8-bit Unicode Transformation Format) is a way of transforming Unicode code points into byte sequences . Its key feature is that it uses variable length : some characters occupy 1 byte, while others require 2, 3, or 4 bytes. This allows for compact texts with simple Latin characters , but it can also represent any character from the Unicode repertoire.

It is fully compatible with ASCII : the first 128 characters (U+0000 to U+007F) are encoded with a single byte identical to the 7-bit ASCII. This facilitated the transition from older systems and explains much of its success on the Internet, in email, and in IETF protocols.

UTF-8 stands out for its robustness : it incorporates synchronization bits that allow for the reliable identification of the start of each symbol. This self-synchronization property makes it easy to detect if a sequence "looks" like UTF-8 , which is very useful in tools and parsers.

Unicode: The Foundation of Everything

Unicode is the universal standard that assigns a unique number to each character , regardless of language, platform, or application. This number is called a code point and is usually written in hexadecimal in the format U+XXXX (or more digits if needed).

For instance, The capital letter “A” is U+0041In HTML we can also refer to it as A. Your computer doesn't "think" of A as a letter, but as the number 65, and then the encoding (such as UTF-8) decides how to represent that number in bytes.

If you want to see how Unicode translates to characters on your PC , on Windows you can hold down the Alt key and type the decimal numeric code on the numeric keypad: for example, Alt+65 returns “A” (see the full list of Alt codes ). It's a classic shortcut that shows how the codes underlie the characters you see.

A bit of history: how UTF-8 was born

UTF-8 was devised by Ken Thompson under the guidance of Rob Pike on September 2, 1992. They implemented it in the Plan 9 operating system from Bell Labs and officially presented it at USENIX (San Diego, January 1993) . During its standardization, sponsored by the X/Open Joint Internationalization Group (XOJIG) , it was known by names such as FSS/UTF and UTF-2 before being consolidated as UTF-8.

The design solved practical problems that plagued previous attempts at universal encoding: ASCII compatibility, self-synchronization, no overlapping bytes, and ease of error detection. This balance made it the de facto standard of the web.

How UTF-8 Works Under the Hood

UTF-8 groups characters according to the bytes needed to encode them . The number of bytes depends exclusively on the Unicode code point and follows bit patterns that indicate the length of the sequence.

• 1 byte (U+0000 to U+007F): ASCII characters. Format: 0xxxxxxx. The most significant bit is 0, which guarantees direct compatibility with ASCII.
• 2 bytes (U+0080 to U+07FF): Format 110yyyyy 10xxxxxx. It is used for most European alphabets with diacritics and others such as Greek, Cyrillic, Hebrew or Arabic..
• 3 bytes (U+0800 to U+FFFF): Format 1110zzzz 10yyyyyy 10xxxxxx. Includes the multilingual basic plan (BMP), with CJK (Chinese, Japanese, Korean), technical symbols and most commonly used characters.
• 4 bytes (U+10000 to U+10FFFF): Format 11110uuu 10uuzzzz 10yyyyyy 10xxxxxx. Represents the supplementary planes: advanced mathematical symbols, historical writings, less common ideographic symbols, etc.

The key to self-synchronization lies in the header bits : 0 for ASCII; 110 for two bytes; 1110 for three; 11110 for four. Continuation bytes always begin with 10. Thanks to this, a continuation byte can never appear as a starting byte , and a valid sequence can never be a substring of a longer one (non-overlap principle).

Equivalence with UTF-16 and surrogate pairs

UTF-16 represents BMP code points with a unit of 16 bits and points above U+FFFF with substitute pairs in the range D800–DFFF. Instead, UTF-8 always encodes real code points, not UTF-16 units, which avoids confusion with substitutes.

Historically, some drafts allowed 5 or 6 bytes in UTF-8 to cover a wider range, but Unicode and RFC 3629 limit UTF-8 to a maximum of 4 bytes . ISO/IEC considered wider options, but these are not part of the current standard.

Practical example: the ñ

The character “ñ” has code point U+00F1, which falls within the two-byte range. Following the pattern, it is encoded as 110xxxxx 10xxxxxx. Its UTF-8 representation is 0xC3 0xB1Decoding is the reverse process: reading the useful bits and reconstructing the original code point.

Advantages and limitations of UTF-8

Main advantages :

• ASCII support: ASCII texts are valid in UTF-8 without changes.
• Universal: can represent any Unicode character, including technical symbols and emojis.
• Efficiency in Latin texts: when using 1 byte for ASCII, saves space compared to UTF-16 in many Western languages.
• Self-synchronization and detection: bit patterns allow detect character beginnings and validate sequences with ease.

Limitations and trade-offs :

• CJK texts take up more space than UTF-16 texts, where many of those characters fit into 2 fixed bytes.
• Computation cost: being of variable length, some operations (e.g., “go to character n”) require going through from the beginning, and certain tasks may be faster in UTF-16/UTF-32.

BOM (Byte Order Mark) in UTF-8

UTF-8 does not need BOM because the byte order does not change the meaning of the values ​​(the minimum unit is the byte). Even so, There is an optional BOM, the character U+FEFF encoded as EF BB BF at the start of a file or stream, which can be used to indicate “this is Unicode/UTF-8”.

Best practices : If it appears at the beginning, some systems accept it and others treat it literally. In concatenations, it's advisable to remove intermediate BOMs . Including it is not mandatory, and its usefulness in UTF-8 is limited compared to UTF-16/UTF-32, where it does mark endianness.

Typical coding errors and how to deal with them

A robust UTF-8 decoder must either reject malformed sequences or replace them with U+FFFD (REPLACEMENT CHARACTER) or flag an error. The most common faults are:

• Truncated sequences: a multibyte lead byte without sufficient continuations.
• Loose continuation bytes: appear 10xxxxxx without a valid lead byte.
• Overlengths: encoding with more bytes than necessary; for example, trying to encode an ASCII with 2 bytes (0xC0 y 0xC1 are invalid).
• Prohibited lengths: starts suggesting 5 or 6 bytes (0xF8 0xFD are not valid in standard UTF-8).
• Values ​​out of Unicode range: not supported above U+10FFFF; certain values ​​(0xF5 0xF7 as beginnings) are invalid.
• UTF-16 surrogate pairs: D800–DFFF are not valid code points in Unicode; they should not appear encoded in UTF-8.

When you see the character “�” on the screen , it's most likely due to an encoding mismatch or a file saved in a different code page. The solution is to force end-to-end UTF-8 encoding (file, server, database, HTTP headers).

UTF-8 on the web and in email

An HTML page only needs to declare one encoding . The recommended encoding, for compatibility and scope, is UTF-8. Include the following meta tag in the header as soon as possible:

<meta charset="UTF-8">

Place it at the beginning of the `<head>` tag so the browser reads it before processing the document. This prevents inconsistencies and "broken" characters. The adoption of UTF-8 on the web is overwhelming ; it's used by the vast majority of current websites.

In email, UTF-8 is widely supported and recommended by organizations such as the Internet Mail Consortium. Configuring email clients to use UTF-8 reduces problems when exchanging messages with people who speak other languages.

UTF-8, UTF-16, and UTF-32: What's the difference?

UTF-8 : Variable-length encoding in 8-bit units; ideal for the web , very efficient with ASCII and Western languages. Excellent compatibility and error detection.

UTF-16 : variable length in 16-bit units; uses surrogate pairs for U+10000 and above. It is often advantageous when non-ASCII characters predominate , and is used in many APIs and platforms (for example, Windows runs natively in UTF-16 ).

UTF-32 : fixed length of 32 bits per character; very simple to index , but space-intensive. It is reserved for cases where size is secondary to simplicity of processing.

Incompatible variants: CESU-8 and “Modified UTF-8”

CESU-8 encodes UTF-16 units directly (including surrogate pairs) instead of encoding code points, thus differing from standard UTF-8 for characters above U+FFFF. Some historical platforms used it: Oracle 8 offered it under the alias UTF-8, and starting with Oracle 9, it added standard UTF-8 under a different alias. Java and Tcl have used CESU-8 in certain contexts.

Modified UTF-8 (for example, in Java environments) represents the NUL character (U+0000) as 0xC0 0x80 instead of 0x00. It avoids the null byte in C strings but is not compliant with the UTF-8 standard. Many implementations of this "modified" version also comply with CESU-8.

UTF-8 on Windows and APIs: Code Pages and Conversion

Windows works internally in UTF-16 (WCHAR), but since Windows 10 version 1903 you can force UTF-8 as the process code page via the application manifest (property activeCodePage). This makes it easier for legacy code that uses “-A” APIs to work over UTF-8.

APIs -A vs -W: The -A depend on the ANSI code page configured (can be CP_UTF8), while the -W they use UTF-16. To interoperate, MultiByteToWideChar y WideCharToMultiByte allow you to convert between UTF-8 and UTF-16; uses CP_UTF8 and, if applicable, MB_ERR_INVALID_CHARS to detect input errors.

Real compatibility: browsers and systems

UTF-8 is compatible with modern browsers (Chrome, Firefox, Safari, Edge, Opera, and recent versions of Internet Explorer) and most operating systems (Windows, Linux, macOS, Android, iOS). Unless you have very old software, you shouldn't have any problems.

How to convert files to UTF-8

In Windows (Notepad) : open the file, go to "File > Save As…", and in "Encoding" choose UTF-8 . Save with a new name if you want to keep the original.

In macOS (TextEdit) : In “TextEdit > Preferences > Open & Save”, select Unicode (UTF-8) when saving. Then export the file with that option enabled.

On Linux: with the terminal you can use iconv. For example: uterine iconv -f <codificación_origen> -t UTF-8 <entrada> -o <salida>. Check later that the application consuming it also expects UTF-8.

How can you tell if a file is in UTF-8? Many modern editors indicate this in the status bar. If you see strange characters like “�”, corrupted accents, or incorrectly displayed “ñ/ç”, check the file encoding and the editor/server/database settings.

Good practices to avoid surprises

Declare UTF-8 as early as possible in HTML and HTTP headers. Align the encoding throughout the entire stack (source files, templates, database, and connection). Avoid mixing encodings on the same page or flow, and use tools that validate/normalize input.

For integrations and APIs, always specify the encoding in headers (Content-Type: application/json; charset=UTF-8, For example). Test with multilingual data (accents, CJK, emojis) to detect weak points before production.

UTF-8 prevailed because it balances compatibility, efficiency, and reach . It is the most practical way to ensure that text travels intact across cultures, systems, and applications, regardless of whether it includes accents, technical symbols, or non-Latin scripts.