Hubbry Logo
Code pointCode pointMain
Open search
Code point
Community hub
Code point
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Contribute something
Code point
Code point
Code point
View on Wikipedia
from Wikipedia

A code point, codepoint or code position is a particular position in a table, where the position has been assigned a meaning. The table may be one dimensional (a column), two dimensional (like cells in a spreadsheet), three dimensional (sheets in a workbook), etc... in any number of dimensions.

Technically, a code point is a unique position in a quantized n-dimensional space, where the position has been assigned a semantic meaning. The table has discrete (whole) and positive positions (1, 2, 3, 4, but not fractions).

Code points are used in a multitude of formal information processing and telecommunication standards.[1][2] For example ITU-T Recommendation T.35[3] contains a set of country codes for telecommunications equipment (originally fax machines) which allow equipment to indicate its country of manufacture or operation. In T.35, Argentina is represented by the code point 0x07, Canada by 0x20, Gambia by 0x41, etc.

In character encoding

[edit]

Code points are commonly used in character encoding, where a code point is a numerical value that maps to a specific character. In character encoding code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but sometimes represent symbols, control characters, or formatting.[4] The set of all possible code points within a given encoding/character set make up that encoding's codespace.[5][6]

For example, the character encoding scheme ASCII comprises 128 code points in the range 0hex to 7Fhex, Extended ASCII comprises 256 code points in the range 0hex to FFhex, and Unicode comprises 1,114,112 code points in the range 0hex to 10FFFFhex. The Unicode code space is divided into seventeen planes (the basic multilingual plane, and 16 supplementary planes), each with 65,536 (= 216) code points. Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112.

In Unicode

[edit]

For Unicode, the particular sequence of bits is called a code unit – for the UCS-4 encoding, any code point is encoded as 4-byte (octet) binary numbers, while in the UTF-8 encoding, different code points are encoded as sequences from one to four bytes long, forming a self-synchronizing code. See comparison of Unicode encodings for details. Code points are normally assigned to abstract characters. An abstract character is not a graphical glyph but a unit of textual data. However, code points may also be left reserved for future assignment (most of the Unicode code space is unassigned), or given other designated functions.[citation needed]

The distinction between a code point and the corresponding abstract character is not pronounced in Unicode but is evident for many other encoding schemes, where numerous code pages may exist for a single code space.[citation needed]

History

[edit]

The concept of a code point dates to the earliest standards for digital information processing and digital telecommunications.

In Unicode, code points are part of Unicode's solution to a difficult conundrum faced by character encoding developers in the 1980s.[7] If they added more bits per character to accommodate larger character sets, that design decision would also constitute an unacceptable waste of then-scarce computing resources for Latin script users (who constituted the vast majority of computer users at the time), since those extra bits would always be zeroed out for such users.[8] The code point avoids this problem by breaking the old idea of a direct one-to-one correspondence between characters and particular sequences of bits.

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Code point

View on Grokipedia
from Grokipedia
A code point is any of the 1,114,112 numerical values in the codespace, ranging from 0 to 10FFFF in notation (U+0000 to U+10FFFF), each potentially assigned to represent an abstract character in the Unicode Standard. These values form the foundation of Unicode's model, distinguishing between assigned code points—such as those for graphic characters, format characters, control characters, and private-use areas—and unassigned or reserved ones like (U+D800–U+DFFF) and noncharacters. In practice, code points are denoted using the "U+" prefix followed by four to six hexadecimal digits, such as U+0041 for the Latin capital letter A, emphasizing their role in abstracting characters from specific glyphs or visual forms to enable universal text representation across scripts and languages. This notation facilitates precise referencing in standards, software, and documentation, where 159,801 characters have been assigned as of 17.0, supporting 172 scripts. The encoding model transforms code points into sequences of code units via defined encoding forms: (variable 1–4 bytes, backward-compatible with ASCII), UTF-16 (1 or 2 16-bit units, using surrogate pairs for code points beyond U+FFFF), and UTF-32 (fixed 32-bit units for direct mapping). This structure allows efficient storage and processing of text, with code points organized into 17 planes—the Basic Multilingual Plane (BMP, U+0000 to U+FFFF) containing most commonly used characters, and supplementary planes for rarer scripts and symbols. Code points may combine in sequences to form grapheme clusters or normalized forms, ensuring compatibility in rendering and collation, while reserved areas prevent conflicts in future expansions of the standard.

Fundamentals

Definition

A code point is a numerical index or value, typically an integer, used in a character encoding scheme to uniquely identify an abstract character from a predefined repertoire. Code points serve as mappings between human-readable characters and machine-readable , enabling the systematic representation and processing of text in systems. For example, in the ASCII encoding, the code point 65 represents the uppercase letter 'A'. In , the same character is identified by the code point U+0041. A code point designates an abstract character, which is a semantic unit independent of its specific encoded form or visual appearance.

Relation to Characters and Glyphs

An abstract character represents a unit of information with semantic value, serving as the smallest component of independent of any particular encoding scheme or visual rendering method. For example, the abstract character for the letter '' embodies the concept of an 'e' with an , regardless of how it is stored digitally or displayed. In the Standard, each such abstract character is uniquely identified by a code point, which is a non-negative assigned within the Unicode codespace. In distinction to abstract characters, a is the particular visual image or shape used to depict a character during rendering or . Glyphs are defined by font technologies and can differ widely; for instance, the abstract character 'A' might be rendered as a glyph in one or a variant in another. The Standard specifies that glyphs are not part of the encoding model itself but result from the interpretation of code points by rendering engines. A code point generally maps to a single abstract character, but the transition from abstract character to introduces variability based on context and presentation rules. One abstract character may correspond to multiple glyphs, such as in the case of positional forms in cursive scripts like , where the shape adapts to , medial, or final positions. Conversely, ligatures can combine multiple abstract characters—each with its own code point—into a single composite , as seen with 'fi' forming a unified shape in many fonts to improve readability. Unicode normalization addresses scenarios where distinct code point sequences encode the same abstract character, enabling consistent text processing across systems. For example, the precomposed 'é' (U+00E9) is canonically equivalent to the sequence 'e' (U+0065) followed by a combining acute accent (U+0301), allowing normalization forms like NFC (which favors precomposed characters) or NFD (which decomposes them) to standardize representations without altering semantic meaning. This equivalence ensures that applications can interchange text reliably while preserving the underlying abstract character.

Representation in Encodings

Code Units vs. Code Points

In character encodings, a code unit represents the smallest fixed-size unit of storage or transmission for text data, typically defined by the bit width of the encoding form. For instance, employs 8-bit code units (bytes), while uses 16-bit code units. These code units serve as the basic building blocks for representing sequences of text, allowing computers to process and interchange data efficiently across different systems. The primary distinction between code points and code units lies in their roles and granularity: a code point is a single numerical value (from 0 to 10FFFF in ) assigned to an abstract character in the standard, whereas code units are the encoded bits that collectively form one or more code points. In fixed-width encodings like UTF-32, each code point corresponds directly to one code unit (a 32-bit value), simplifying access. However, in variable-width encodings such as and UTF-16, a single code point often requires multiple code units to encode, particularly for characters beyond the Basic Multilingual Plane. This multi-unit representation enables compact storage but introduces complexity in parsing text streams. A concrete example illustrates this difference: the Unicode code point U+1F600, which maps to the grinning face emoji (😀), is encoded as four 8-bit code units in UTF-8 (hexadecimal F0 9F 98 80, or bytes 240, 159, 152, 128) and as two 16-bit code units in UTF-16 (hexadecimal D83D DE00, forming a surrogate pair). In UTF-16, the first unit (D83D) is a high surrogate and the second (DE00) a low surrogate, together representing the full code point; treating them separately would yield invalid or unintended characters. When processing text, algorithms must correctly decode sequences of code units into complete code points to ensure accurate interpretation of abstract characters. Failure to handle multi-unit code points properly—such as by assuming each code unit is an independent character—can result in errors like , where encoded text is misinterpreted and rendered as garbled symbols during decoding with an incompatible scheme. This underscores the need for encoding-aware software to normalize and validate input, preventing in applications ranging from web browsers to file systems.

Fixed-Width Encodings

Fixed-width encodings are schemes in which each code point is represented using a consistent number of code units, such as bits or bytes, resulting in sequences of uniform length for all characters. This direct one-to-one mapping between code points and fixed-size code units simplifies the representation process, as no variable-length sequences are required to encode different characters. The primary advantages of fixed-width encodings include ease of implementation and processing, since there is no need for complex decoding algorithms to determine character boundaries. They also enable efficient to individual characters within a text stream, allowing the position of the nth character to be computed in constant time by simple arithmetic on the code unit offsets. These properties make fixed-width encodings particularly suitable for applications with small character repertoires, where simplicity outweighs storage efficiency. However, fixed-width encodings have significant limitations due to their uniform sizing, which caps the total number of representable code points at the power of two corresponding to the width (e.g., 128 for 7 bits or 256 for 8 bits). This restricts their ability to accommodate large or diverse character sets, such as those required for multilingual text, often necessitating multiple incompatible for different languages. For the full range, UTF-32 is a fixed-width encoding using 32-bit code units, providing direct mapping for all 1,114,112 possible code points without surrogates or variable lengths, though it uses more storage for ASCII-range text compared to variable-width forms. Prominent examples include ASCII, a 7-bit encoding supporting 128 code points from 0x00 to 0x7F, primarily for English-language text and control characters. ISO/IEC 8859-1, an 8-bit extension of ASCII, provides 256 code points for Western European languages, with the first 128 matching ASCII. , another 8-bit scheme developed by , uses a different bit assignment for characters and remains in use on mainframe systems. , a variant of ISO/IEC 8859-1, also employs 8 bits but includes additional printable characters in the upper range for enhanced Western European support.

Variable-Width Encodings

Variable-width encodings represent Unicode code points using a varying number of code units, allowing for more efficient storage of text with predominantly low-range characters while supporting the full range up to U+10FFFF. This approach contrasts with fixed-width encodings like UTF-32, which allocate uniform space regardless of the code point value. By adjusting the number of code units based on the code point's magnitude, these encodings optimize space for common scripts such as Latin and Cyrillic, which fit into fewer units, while extending to rarer or higher-range characters with additional units. UTF-8, a widely used variable-width encoding, employs 8-bit code units and determines the sequence length from the leading bits of the first byte. Code points in the range U+0000 to U+007F (basic Latin and ASCII) are encoded in a single byte, ensuring compatibility with legacy ASCII systems. For U+0080 to U+07FF (e.g., extended Latin, Greek, Cyrillic, Arabic), two bytes are used; U+0800 to U+FFFF (including most of the Basic Multilingual Plane, or BMP) require three bytes; and U+10000 to U+10FFFF (supplementary planes) use four bytes. The encoding algorithm ensures that continuation bytes (always 10xxxxxx in binary) follow the lead byte, which specifies the total length, enabling a self-synchronizing property where parsers can detect sequence boundaries efficiently, even after data corruption, by examining at most four bytes backward.
Code Point RangeBytes in UTF-8Example Characters
U+0000–U+007F1Basic Latin (A–Z)
U+0080–U+07FF2Extended Latin, Greek, Cyrillic, Arabic
U+0800–U+FFFF3Devanagari, Thai, BMP Han
U+10000–U+10FFFF4Emoji, supplementary CJK
UTF-16 uses 16-bit code units and encodes most code points in the BMP (U+0000 to U+FFFF) with a single unit, making it compact for scripts like European languages and many Asian ideographs. For code points beyond U+FFFF, it employs surrogate pairs: a high surrogate (U+D800 to U+DBFF) followed by a low surrogate (U+DC00 to U+DFFF), forming a two-unit (four-byte) sequence that represents one supplementary code point. This mechanism reserves 2,048 code points in the BMP for surrogates, ensuring reversible mapping without overlap. These encodings balance space efficiency and implementation complexity for large character sets. excels in storage for predominantly ASCII or European text, where over 90% of bytes may be single-unit in English documents, but expands significantly for East Asian scripts, potentially using three or four bytes per character. offers better performance for BMP-heavy content in processing environments like or Windows APIs, as most operations avoid surrogate handling, though its variable length introduces parsing overhead compared to fixed-width alternatives. Both require careful byte-order handling (e.g., via BOM) to avoid misinterpretation across .

Code Points in Unicode

Unicode Range and Allocation

The Unicode Standard defines a total of 1,114,112 code points, ranging from U+0000 to U+10FFFF, organized into 17 planes each containing 65,536 code points. This expansive codespace accommodates the encoding of characters from all known writing systems while leaving substantial room for expansion. As of Unicode 17.0, released in September 2025, approximately 159,801 code points are assigned to characters, with the remainder categorized as reserved, unallocated, or designated as noncharacters. Reserved code points include areas for (2,048 code points in the range U+D800–U+DFFF), private use (137,468 code points in the private use areas across the Basic Multilingual Plane and Planes 15 and 16), ensuring specific functions without assignment to abstract characters. Unallocated code points remain available for future assignments, while noncharacters—totaling 66—such as U+FDD0–U+FDEF (32 points) and the pairs U+FFFE and U+FFFF in each plane (34 points)—are explicitly not used for and serve purposes like end-of-text markers or byte-order detection. These categories maintain a clear distinction between usable character space and specialized reservations. The allocation principles of Unicode emphasize stability, universality, and future-proofing. Stability is enforced through policies that prohibit the reallocation, removal, or modification of assigned code points, ensuring that once encoded, a character's semantics and properties remain unchanged across versions. Universality aims to encompass characters from every worldwide, harmonized with ISO/IEC 10646 to support global without bias toward any or script. Future-proofing is achieved by allocating only a fraction of the codespace initially, preserving over 95% as unallocated to accommodate unforeseen needs, such as emerging scripts or extensions, without disrupting existing implementations. Recent updates reflect these principles without altering the overall range. 17.0 added 4,803 new code points, including characters for four new scripts (such as Beria Erfe and Sidetic) and eight new , bringing the total assigned characters to 159,801 while adhering to stability rules and reserving space for future growth. No major changes to the codespace boundaries or allocation categories have occurred by late 2025.

Planes, Blocks, and Surrogates

Unicode code points are organized into 17 planes, each comprising 65,536 consecutive code points, to facilitate the encoding of a vast repertoire of characters while maintaining compatibility with earlier standards. Plane 0, known as the Basic Multilingual Plane (BMP), spans U+0000 to U+FFFF and contains the most commonly used characters from virtually all modern writing systems, including Latin, Greek, Cyrillic, and many others, ensuring that legacy systems can handle a significant portion of Unicode without modification. Planes 1 through 16 extend the codespace for less common or specialized scripts; for instance, Plane 2, the Supplementary Ideographic Plane (SIP), accommodates additional CJK Unified Ideographs from U+20000 to U+2FFFF, supporting expanded needs for East Asian typography. Within these planes, the Unicode codespace is further subdivided into blocks, which group code points thematically by script, symbol category, or functional purpose, aiding in the organization and lookup of characters. Each block typically ranges from 16 to 256 code points, though sizes vary, and unassigned areas exist for future allocations. For example, the Basic Latin block from U+0000 to U+007F includes the 128 characters of the ASCII standard, such as letters, digits, and , forming the foundation for English and many Western languages. To represent code points beyond the BMP in UTF-16 encoding, surrogate code points in Plane 0 are reserved for pairing to access the supplementary planes (1–16). These consist of 2,048 high surrogates from U+D800 to U+DBFF and 2,048 low surrogates from U+DC00 to U+DFFF, forming 1,048,576 possible pairs that map to code points U+10000 through U+10FFFF. In UTF-16, a supplementary code point is thus encoded as a 32-bit sequence: a 16-bit high surrogate followed by a 16-bit low surrogate, calculated such that the high surrogate's offset (U+D800 subtracted) combined with the low surrogate's (U+DC00 subtracted) yields the supplementary value. Software implementations must properly decode these surrogate pairs to retrieve the full code point; unpaired surrogates or mismatched pairs are considered invalid and typically trigger errors or replacement with substitution characters to maintain .

Historical Development

Pre-Unicode Encodings

The concept of code points originated in early mechanical and electrical systems for representing discrete symbols through numerical assignments. In the late , punched cards emerged as a medium for encoding data, with Herman Hollerith's system for the 1890 U.S. Census using patterns of holes to denote numerical values from 0 to 9, effectively assigning code points to digits for tabulation purposes. This approach laid foundational principles for mapping symbols to fixed positions in a code set, primarily for numeric data but extending to alphabetic characters for comprehensive census tabulation. Telegraph systems in the same era advanced further. The , patented by in 1874 and based on his 1870 invention, employed a 5-bit binary scheme to define 32 distinct code points, primarily for uppercase letters, numbers, and control signals in asynchronous transmission over teleprinters. This 5-bit limitation reflected hardware constraints of the time, such as mechanical keyboards and early electrical relays, but it enabled efficient of multiple channels on telegraph lines. By the 1960s, computing demanded broader standardization for text interchange. The American Standard Code for Information Interchange (ASCII), developed through collaboration among telecommunications and computer manufacturers, was published as ASA X3.4-1963 by the American Standards Association (predecessor to ANSI), defining a 7-bit code with 128 code points to encompass English uppercase and lowercase letters, digits, , and control characters. This fixed-width encoding prioritized compatibility with existing teletype equipment while allocating the upper bit for parity or future extensions, marking a shift toward universal adoption in U.S.-centric systems. The 1970s and 1980s saw proliferation of 8-bit extensions to address non-English scripts, introducing the notion of code pages as variant mappings within the expanded 256 code points. The ISO/IEC 8859 series, first published in 1987 by the , comprised multiple parts tailored to regional needs, such as ISO/IEC 8859-1 (Latin-1) for Western European languages including accented characters beyond ASCII. These standards extended the 7-bit ASCII subset into the upper 128 positions for language-specific glyphs, facilitating adoption in personal computers and international . Despite these advances, pre-Unicode encodings suffered from profound incompatibilities, as each system optimized for local languages without global coordination. For instance, , devised in 1984 by Taiwan's , used variable-width bytes to encode over 13,000 but overlapped ambiguously with ASCII ranges, rendering it incompatible with Japanese encodings like . , developed in 1982 by and for Japanese , , and hiragana, similarly prioritized single- and double-byte sequences tailored to East Asian scripts, leading to data corruption—known as —when texts were exchanged across platforms. Such fragmentation, driven by national standards and vendor-specific implementations, resulted in hundreds of rival code sets and hindered cross-border digital communication.

Unicode Evolution

The development of the Unicode standard originated in late 1987, when engineers Joe Becker from , Lee Collins from Apple, and Mark Davis from Taligent initiated discussions to create a universal system capable of supporting multiple writing systems beyond the limitations of ASCII. This effort addressed the fragmentation of existing encodings for different languages and scripts, aiming for a single, unified approach to text representation. By 1988, the project had formalized a character database, and in 1989, it expanded to include collaborators from organizations such as and the Research Libraries Group, aligning the scope with emerging international standards. In 1990, joined the initiative, and work focused on mapping to the draft ISO/IEC 10646 standard, finalizing for East Asian ideographs, and establishing a compatibility zone for legacy encodings. The was officially incorporated in on January 3, 1991, to oversee the project. That year, the Consortium merged efforts with ISO/IEC 10646, agreeing to maintain full compatibility between and the emerging ISO standard, which ensured synchronized growth and avoided divergent paths in global text encoding. 1.0 was released in October 1991, defining an initial repertoire of over 7,000 code points primarily in the Basic Multilingual Plane (BMP), covering major scripts like Latin, Greek, Cyrillic, , and Hebrew, while prioritizing with ASCII in the range U+0000 to U+007F. Subsequent versions built on this foundation, expanding the encoded repertoire to support globalization. 2.0, released in , significantly broadened the character set by incorporating additional scripts and completing the initial allocation plan for the BMP, which spans 65,536 code points from U+0000 to U+FFFF and serves as the core plane for common usage. This version added support for scripts such as Armenian, Georgian, and Thai, enhancing compatibility with regional standards. By 3.1 in , the standard began populating Plane 1 (the Supplementary Multilingual Plane, U+10000 to U+1FFFF), introducing historic scripts like Gothic, Deseret, and Old Italic, marking the first extensions beyond the BMP to accommodate less frequently used or ancient writing systems. Unicode 6.0, released in 2010, further solidified the standard's maturity amid rising , coinciding with the widespread dominance of as the preferred encoding for due to its ASCII compatibility and variable-length . This version added thousands of characters for scripts like Lepcha and Vai, while 's adoption exceeded 90% of web pages by the mid-2010s, driven by browser support and XML standards. integration emerged around 2007, with initial symbol additions evolving into dedicated emoji support; by Unicode 5.2 in 2009, the first explicitly emoji-intended characters were encoded for cross-platform , addressing the need for visual expression in digital communication. In the modern era, Unicode has adopted an annual release cycle to accommodate ongoing demands for script inclusion and cultural representation. For instance, Unicode 14.0 in 2021 added 838 characters, primarily additions to existing scripts and symbols, including new emoji and extensions for historical notations, bringing the total assigned code points to 144,697. Unicode 16.0, released in 2024, introduced 5,185 new characters—such as the West African Garay script and historic Tulu-Tigalari—along with enhancements for and legacy symbols, resulting in a total of 154,998 assigned code points. Unicode 17.0, released in September 2025, added 4,803 characters, including four new scripts such as Beria Erfe and Sidetic, along with emoji and symbol extensions, bringing the total assigned code points to 159,801.

References

  1. https://www.unicode.org/glossary/
  2. https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-2/
  3. http://www.unicode.org/standard/principles.html
  4. https://www.w3.org/International/articles/definitions-characters/
  5. http://www.unicode.org/reports/tr17/tr17-3.html
  6. https://www.columbia.edu/kermit/ascii.html
  7. https://developer.mozilla.org/en-US/docs/Glossary/Code_point
  8. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-2/
  9. https://unicode.org/reports/tr15/
  10. http://www.unicode.org/reports/tr17/
  11. https://www.fileformat.info/info/unicode/char/1f600/index.htm
  12. https://www.unicode.org/reports/tr17/tr17-2.html
  13. https://cs.lmu.edu/~ray/notes/charenc/
  14. https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0244r2.html
  15. https://www.iso.org/standard/28245.html
  16. https://www.ibm.com/docs/en/zos-basic-skills?topic=mainframe-ebcdic-character-set
  17. https://learn.microsoft.com/en-us/windows/win32/intl/code-page-identifiers
  18. https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-1/
  19. https://www.unicode.org/versions/Unicode17.0.0/
  20. https://www.unicode.org/reports/tr44/
  21. https://www.unicode.org/policies/stability_policy.html
  22. https://blog.unicode.org/2025/09/unicode-170-release-announcement.html
  23. https://www.unicode.org/charts/
  24. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/
  25. https://www.ibm.com/history/punched-card
  26. https://homepage.divms.uiowa.edu/~jones/cards/history.html
  27. https://www.ece.unb.ca/tervo/ee4253/baudot.shtml
  28. https://www.smithsonianmag.com/smart-news/roots-computer-code-lie-telegraph-code-180964782/
  29. https://www.aivosto.com/articles/charsets-7bit.html
  30. https://learn.sparkfun.com/tutorials/ascii/history
  31. https://cdn.standards.iteh.ai/samples/16338/9e2f3aa895034be18a3105fe31d71f33/ISO-8859-1-1987.pdf
  32. https://www.iso.org/standard/16344.html
  33. https://www.cns11643.gov.tw/pageView.jsp?ID=9&la=1
  34. https://www.edrdg.org/~jwb/paperdir/kanjicomp.html
  35. http://www.cip.ifi.lmu.de/~bolzer/unicode_intro/
  36. https://www.unicode.org/history/summary.html
  37. https://www.unicode.org/versions/enumeratedversions.html
  38. http://www.unicode.org/reports/tr27/tr27-1.html
  39. https://unicode.org/reports/tr51/
  40. http://blog.unicode.org/2021/09/announcing-unicode-standard-version-140.html
  41. https://www.unicode.org/versions/Unicode16.0.0/
Add your contribution
Related Hubs
Contribute something
User Avatar
No comments yet.