- Unicode assigns a unique code point to each character and synchronizes its repertoire with ISO/IEC 10646.
- UTF-8, UTF-16, and UTF-32 encode the same characters and allow lossless conversion.
- Normalization, Bidi, and UCD properties ensure consistent comparison, ordering, and rendering.
- Adopting Unicode (preferably UTF-8 on the web) prevents corruption and facilitates internationalization.
Unicode is the common language that computers speak when handling text : it assigns a unique number to each character and symbol, from almost any writing system, so that it can be stored, processed, and shared seamlessly between platforms and countries.
This standard emerged to overcome the limitations of old character sets ( the ASCII set , code pages, EBCDIC, etc.), which fell short or were incompatible with each other, and today it is synchronized with ISO/IEC 10646, maintains algorithms (such as the bidirectional one) and defines properties and normalization rules so that everything works consistently.
What is Unicode and why is it so important?
Unicode is a universal and constantly evolving character encoding standard that describes each character with a name, a code point, and a set of properties (script, category, directionality, case, etc.). The Unicode Technical Committee (UTC), within the Unicode Consortium, keeps it in sync with the ISO/IEC 10646 standard.
Its goal is universality, uniformity, and uniqueness : a broad repertoire that allows for the unambiguous exchange of multilingual text, with clear and reproducible rules. Thanks to this, modern technologies (operating systems, browsers, XML, Java, databases) can mix Latin and Arabic scripts, CJK ideograms, emojis, and technical or musical symbols within the same document.
Characters, glyphs and code points
In Unicode, a character is an abstract unit of information, and a code point is its numerical identifier . A glyph is the visual form (what you see on screen), which depends on the font . A single character can have multiple glyphs, and sometimes a glyph represents more than one character.
The usual notation for a code point is U+XXXX in hexadecimal . Illustrative examples from the analyzed material: the lowercase 'l' is U+006C, the pre-composed lowercase 'ü' is U+00FC, the 'é' is U+00E9 and the Greek beta: the uppercase is U+0392 and the lowercase U+03B2 (in some texts 'β' is cited with U+0392, but that code corresponds to the uppercase 'Β').
The Unicode code space has 1.114.112 possible positions (up to U+10FFFF) , organized to cover and classify all writing systems and symbols, with tens of thousands of effective and growing assignments in each version.
Plans, areas and blocks
Unicode divides its character space into 17 planes of up to 65.536 code points each . This organization makes it easy to group related scripts and symbols and quickly find what you're looking for.
Most relevant planes : the Basic Multilingual Plane (BMP, plane 0) brings together almost all modern alphabets and many symbols; the Supplementary Multilingual Plane (SMP, plane 1) stores historical scripts and technical symbols (e.g., musical and mathematical); the Supplementary Ideographic Plane (SIP, plane 2) expands the CJK ideograms; plane 14 (SSP) includes special labels; and planes 15 and 16 are for private use.
Blocks and areas : the maps are informally subdivided into areas and formally into contiguous blocks. Blocks are used to tabulate and document characters, although they do not always correspond to meaningful linguistic groupings.
CJK ideograms and the Unihan project
East Asian (Han) ideograms are unified in Unicode with stylistic variations by region , but all refer to the same abstract character. Their management is handled by the Ideographic Rapporteur Group (IRG), an ISO/IEC JTC1/SC2/WG2 group with representation from China, Japan, Korea, Vietnam, Hong Kong, Macau, Singapore, the USA, and others.
The Unihan database brings together auxiliary information (readings, meanings, equivalences) essential for handling these ideograms in different languages and historical or corporate standards.
Main CJK blocks and extensions :
- CJK Unified Ideographs (BMP, U+4E00–U+9FFF): 20.992 commonly used characters.
- Extension A (BMP, U+3400–U+4DBF): 6.592 less frequent ideograms.
- B–H extensions: in SMP/SIP/TIP, they add up to tens of thousands more (B: U+20000–U+2A6DF, 42.720; C: U+2A700–U+2B73F, 4.154; D: U+2B740–U+2B81F, 222; E: U+2B820–U+2CEAF, 5.762; F: U+2CEB0–U+2EBEF, 7.473; G: U+30000–U+3134F, 4.939;
- Other related CJK blocks: Kangxi radicals (U+2F00–U+2FDF), CJK symbols and punctuation (U+3000–U+303F), Compatibility and compatibility formats, Compatibility ideogram supplement, etc.
Unicode anticipates that the incorporation of ideograms will not have an absolute end and contemplates mechanisms such as ideographic description sequences to represent uncoded symbols by breaking them down into existing components (with cautions: without canonical decomposition or guarantees in operations such as searching or ordering).
Encoding forms: UTF-8, UTF-16 and UTF-32
Unicode defines transformation forms (UTF) that convert code points into storage units so that software can efficiently process text according to its context.
UTF-8 is a variable-length, byte-oriented encoding format that is ASCII-compatible in the range U+0000–U+007F in a single byte. The current standard uses 1 to 4 bytes per character; some older texts mention 1–6, but in modern Unicode it is 1–4. It is the dominant format on the web.
UTF-16 uses 16-bit units
(one or two code units per character) : most BMP characters fit into one unit; supplementary characters use surrogate pairs in the range U+D800–U+DFFF.
UTF-32 is fixed length: 4 bytes per character, simple but space-intensive; useful when direct character indexing is important and memory is not an issue.
How bits are distributed in UTF-8
The UTF-8 format distributes the code point bits into sequences of 1 to 4 bytes with recognizable headers, which avoids ambiguities and facilitates the detection of character boundaries.
| Unicode Range | Bit pattern | Bytes |
|---|---|---|
| U+0000..U+007F | 0xxxxxxx | 1 |
| U+0080..U+07FF | 110yyyyy 10xxxxxx | 2 |
| U+0800..U+FFFF | 1110zzzz 10yyyyyy 10xxxxxx | 3 |
| U+010000..U+10FFFF | 11110uuu 10uuuzzzz 10yyyyyy 10xxxxxx | 4 |
A key advantage of UTF-8 is that it avoids byte order (endianness) problems and allows for highly efficient processing of text streams, in addition to being backward compatible with ASCII.
Encoding schemes, endianness, and BOM
In addition to UTF forms, Unicode describes serialization schemes that resolve how bytes are transmitted between systems with different endianness and how aspects such as byte order are signaled.
- UTF-8: endianness does not apply. It can be carried Byte Order Mark (BOM) as a hint, although it is not required or recommended by default.
- UTF-16: BE/LE (big/little-endian) variants and optional BOM (if missing and not defined by the protocol, big-endian is assumed).
- UTF-32: BE/LE variants with analogous rules; BOM allowed as an order mark.
There are also specific variants such as UTF-16BE/UTF-16LE and UTF-32BE/UTF-32LE (without BOM by convention), and other historical compatibility encodings such as UTF-7 or UTF-EBCDIC, in addition to GB18030 (Chinese equivalent to UTF-8 with support for simplified and traditional Chinese).
Normalization, composition and equivalences
Many characters can be represented as pre-composed or as sequences of base + combining marks . Classic examples from the revised material: the pre-composed 'Ä' is U+00C4, while the decomposed form is 'A' (U+0041) + diaeresis (U+0308). The Vietnamese 'ỗ' can be represented as 'o' (U+006F) + circumflex (U+0302) + tilde (U+0303).
Unicode defines standardized forms and two types of equivalence : canonical (same essential content) and compatibility (forms that may appear the same but have semantic differences). Standardization ensures reliable string comparison and reduces duplication.
Bidirectional algorithm and special characters
For right-to-left scripts, such as Arabic or Hebrew, Unicode incorporates a standardized and evolving bidirectional algorithm (Bidi) (e.g., revisions in 6.3), so that mixed texts with Latin and Arabic are represented in the correct visual order.
Classes of code points that should be known according to the received material: graphic characters (letters, signs, symbols), formatting (invisible characters that affect processing: U+2028 line break, U+2029 paragraph break, U+00A0 hard space), inherited control codes for compatibility (ranges U+0000–U+001F, U+007F, U+0080–U+009F), private use, reserved positions, substitutes (U+D800–U+DFFF for UTF-16) and non-characters (U+FFFE, U+FFFF in each plane).
Unicode, ISO/IEC 10646 and other standards (ASCII, ANSI, code pages)
Unicode is synchronized with ISO/IEC 10646 (UCS) and retains mappings to previous standards (ASCII, ISO 8859-1, ANSI Z39.64, JIS X 0208, KS X 1001, GB 2312, GB 18030, HKSCS, CNS 11643, etc.), in addition to reserving spaces for private use by manufacturers.
ASCII vs. Unicode : ASCII is a 7-bit character set with 128 characters , sufficient for basic English, but insufficient for languages with diacritics, ideograms, or emojis . Unicode covers more than 140.000 characters and continues to grow, with 8/16/32-bit encoding according to the UTF-12 format.
ANSI and code pages : 'ANSI' usually refers to limited and mutually incompatible 8-bit Windows code pages. A computer running OEM-Latin II that opens text in IBM EBCDIC-Cyrillic will see incorrect characters. Unicode overcomes this problem with a single character set and lossless conversions between its own forms.
Implementation on systems (Windows, Solaris) and conversion
Windows internally uses UTF-16 for its modern APIs and offers features like MultiByteToWideChar and WideCharToMultiByte for converting between Unicode and code pages (SBCS/DBCS/MBCS). If you are forced to convert to a code page, there may be data loss if it cannot represent all characters.
New applications on Windows should use UTF-16 internally and leave the conversion to the edges (I/O, protocols), minimizing corruption. Even so, Windows maintains compatibility with code pages when unavoidable.
Oracle Solaris 11, according to the revised documentation, supports Unicode 6.0 and ISO/IEC 10646:2011 at the system level, using UTF-8 as the default format in its parameter sets, which saves byte ordering problems and transparently preserves ASCII.
Unicode in Practice: Web, Documents, and Programming
On the web, it's common to serve and store content in UTF-8 . In HTML, declare '<meta charset="UTF-8">' to ensure compatibility, and use hex numeric entities when necessary (example: the euro '€').
In Word, you can easily insert Unicode symbols by placing the cursor, going to Insert > Symbol, or by typing the code and pressing Alt+X; this converts the value to its character, following standard procedures. See also the list of Alt codes for inserting symbols from the keyboard.
In programming languages : in Python 3, strings are already Unicode; in Java and C#, you can use escape characters '\uXXXX'; in HTML, '&#xXXXX;'. The important thing is that editing, compilation, and transmission use the same encoding to avoid errors.
Copying and pasting symbols works if the entire string is in Unicode . If parts of it use a different encoding, question marks, squares, or other strange symbols may appear.
Character database (UCD), properties and categories
The Unicode Character Database (UCD) publishes, for each code point, its name, category, script, case properties, directionality, and other characteristics . This information is vital for rendering engines and text processing to function correctly.
Thanks to these properties , different components and algorithms (such as sorting, word segmentation, or uppercase/lowercase transformation) can behave consistently with the same data.
Versions and extensions of the standard: highlights
Unicode grows with each version , incorporating new scripts, symbols, and emojis. Key milestones include version 1.0 in 1991 with 7.161 characters, and subsequent expansions that have added thousands of new symbols, ideograms, emojis, and scripts.
For example, in 2022 , version 15.0 added 4.192 additional CJK ideograms and other elements, reaching approximately 149.186 characters.
Tools for exploring the Unicode table
There are free utilities for searching and analyzing characters : Unibook Character Browser, SYMBL, and Branah.com allow you to query properties, names, and character blocks, making it easier for developers and content creators to quickly find the information they need.
Use cases: address forms, internationalization, and business
Allowing users to enter addresses in their own language and script helps reduce errors and improve the user experience. Additionally, Unicode interoperability prevents conversion issues between different systems, maintaining text integrity throughout the digital chain.
For this reason, Unicode is the fundamental element that ensures that the text remains final throughout the digital infrastructure , from web forms to database systems and printed documents, regardless of whether you use 'ñ', Arabic, Chinese or emojis in the same sentence.
