Hubbry Logo
Numeric character referenceNumeric character referenceMain
Open search
Numeric character reference
Community hub
Numeric character reference
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Contribute something
Numeric character reference
Numeric character reference
Numeric character reference
View on Wikipedia
from Wikipedia

A numeric character reference (NCR) is a common markup construct used in SGML and SGML-derived markup languages such as HTML and XML. It consists of a short sequence of characters that, in turn, represents a single character. Since WebSgml, XML and HTML 4, the code points of the Universal Character Set (UCS) of Unicode are used. NCRs are typically used in order to represent characters that are not directly encodable in a particular document (for example, because they are international characters that do not fit in the 8-bit character set being used, or because they have special syntactic meaning in the language). When the document is interpreted by a markup-aware reader, each NCR is treated as if it were the character it represents.

Examples

[edit]

In SGML, HTML, and XML, the following are all valid numeric character references for the Greek capital letter Sigma

Numerical character reference of U+03A3 Σ GREEK CAPITAL LETTER SIGMA
(3A316 = 93110)
Unicode character Numerical base Numerical reference in markup Effect
U+03A3 Decimal Σ Σ
U+03A3 Decimal Σ Σ
U+03A3 Hexadecimal Σ Σ
U+03A3 Hexadecimal Σ Σ
U+03A3 Hexadecimal Σ Σ

In SGML, HTML, and XML, the following are all valid numeric character references for the Latin capital letter AE

Numerical character reference of U+00C6 Æ LATIN CAPITAL LETTER AE
Unicode character Numerical base Numerical reference in markup Effect
U+00C6 Decimal Æ Æ
U+00C6 Hexadecimal Æ Æ

In SGML, HTML, and XML, the following are all valid numeric character references for the Latin small letter sharp s ß

Numerical character reference of U+00DF ß LATIN SMALL LETTER SHARP S
Unicode character Numerical base Numerical reference in markup Effect
U+00DF Decimal ß ß
U+00DF Hexadecimal ß ß

List of numeric character references for the printable ASCII characters:

Unicode character Character
Reference
(decimal) 		Character
Reference
(hexadecimal) 		Effect
U+0020     (space)
U+0021 ! ! !
U+0022 " " "
U+0023 # # #
U+0024 $ $ $
U+0025 % % %
U+0026 & & &
U+0027 ' ' '
U+0028 ( ( (
U+0029 ) ) )
U+002A * * *
U+002B + + +
U+002C , , ,
U+002D - - -
U+002E . . .
U+002F / / /
U+0030 0 0 0
U+0031 1 1 1
U+0032 2 2 2
U+0033 3 3 3
U+0034 4 4 4
U+0035 5 5 5
U+0036 6 6 6
U+0037 7 7 7
U+0038 8 8 8
U+0039 9 9 9
U+003A : : :
U+003B &#59; &#x3B; ;
U+003C &#60; &#x3C; <
U+003D &#61; &#x3D; =
U+003E &#62; &#x3E; >
U+003F &#63; &#x3F; ?
U+0040 &#64; &#x40; @
U+0041 &#65; &#x41; A
U+0042 &#66; &#x42; B
U+0043 &#67; &#x43; C
U+0044 &#68; &#x44; D
U+0045 &#69; &#x45; E
U+0046 &#70; &#x46; F
U+0047 &#71; &#x47; G
U+0048 &#72; &#x48; H
U+0049 &#73; &#x49; I
U+004A &#74; &#x4A; J
U+004B &#75; &#x4B; K
U+004C &#76; &#x4C; L
U+004D &#77; &#x4D; M
U+004E &#78; &#x4E; N
U+004F &#79; &#x4F; O
U+0050 &#80; &#x50; P
U+0051 &#81; &#x51; Q
U+0052 &#82; &#x52; R
U+0053 &#83; &#x53; S
U+0054 &#84; &#x54; T
U+0055 &#85; &#x55; U
U+0056 &#86; &#x56; V
U+0057 &#87; &#x57; W
U+0058 &#88; &#x58; X
U+0059 &#89; &#x59; Y
U+005A &#90; &#x5A; Z
U+005B &#91; &#x5B; [
U+005C &#92; &#x5C; \
U+005D &#93; &#x5D; ]
U+005E &#94; &#x5E; ^
U+005F &#95; &#x5F; _
U+0060 &#96; &#x60; '
U+0061 &#97; &#x61; a
U+0062 &#98; &#x62; b
U+0063 &#99; &#x63; c
U+0064 &#100; &#x64; d
U+0065 &#101; &#x65; e
U+0066 &#102; &#x66; f
U+0067 &#103; &#x67; g
U+0068 &#104; &#x68; h
U+0069 &#105; &#x69; i
U+006A &#106; &#x6A; j
U+006B &#107; &#x6B; k
U+006C &#108; &#x6C; l
U+006D &#109; &#x6D; m
U+006E &#110; &#x6E; n
U+006F &#111; &#x6F; o
U+0070 &#112; &#x70; p
U+0071 &#113; &#x71; q
U+0072 &#114; &#x72; r
U+0073 &#115; &#x73; s
U+0074 &#116; &#x74; t
U+0075 &#117; &#x75; u
U+0076 &#118; &#x76; v
U+0077 &#119; &#x77; w
U+0078 &#120; &#x78; x
U+0079 &#121; &#x79; y
U+007A &#122; &#x7A; z
U+007B &#123; &#x7B; {
U+007C &#124; &#x7C; |
U+007D &#125; &#x7D; }
U+007E &#126; &#x7E; ~

Discussion

[edit]

Markup languages are typically defined in terms of UCS or Unicode characters. That is, a document consists, at its most fundamental level of abstraction, of a sequence of characters, which are abstract units that exist independently of any encoding.

Ideally, when the characters of a document utilizing a markup language are encoded for storage or transmission over a network as a sequence of bits, the encoding that is used will be one that supports representing each and every character in the document, if not in the whole of Unicode, directly as a particular bit sequence.

Sometimes, though, for reasons of convenience or due to technical limitations, documents are encoded with an encoding that cannot represent some characters directly. For example, the widely used encodings based on ISO 8859 can only represent, at most, 256 unique characters as one 8-bit byte each.

Documents are rarely, in practice, ever allowed to use more than one encoding internally, so the onus is usually on the markup language to provide a means for document authors to express unencodable characters in terms of encodable ones. This is generally done through some kind of "escaping" mechanism.

The SGML-based markup languages allow document authors to use special sequences of characters from the ASCII range (the first 128 code points of Unicode) to represent, or reference, any Unicode character, regardless of whether the character being represented is directly available in the document's encoding. These special sequences are character references.

Character references that are based on the referenced character's UCS or Unicode code point are called numeric character references. In HTML 4 and in all versions of XHTML and XML, the code point can be expressed either as a decimal (base 10) number or as a hexadecimal (base 16) number. The syntax is as follows:

Character U+0026 (ampersand), followed by character U+0023 (number sign), followed by one of the following choices:

  • one or more decimal digits zero (U+0030) through nine (U+0039); or
  • character U+0078 ("x") followed by one or more hexadecimal digits, which are zero (U+0030) through nine (U+0039), Latin capital letter A (U+0041) through F (U+0046), and Latin small letter a (U+0061) through f (U+0066);

all followed by character U+003B (semicolon). Older versions of HTML disallowed the hexadecimal syntax.

The characters that comprise a numeric character reference can be represented in every character encoding used in computing and telecommunications today, so there is no risk of the reference itself being unencodable.

There is another kind of character reference called a character entity reference, which allows a character to be referred to by a name instead of a number. (Naming a character creates a character entity.) HTML defines some character entities, but not many; all other characters can only be included by direct encoding or using NCRs.

Restrictions

[edit]

The Universal Character Set defined by ISO 10646 is the "document character set" of SGML, HTML 4, so by default, any character in such a document, and any character referenced in such a document, must be in the UCS.

While the syntax of SGML does not prohibit references to invalid or unassigned code points, such as &#xFFFF;, SGML-derived markup languages such as HTML and XML can, and often do, restrict numeric character references to only those code points that are assigned to characters.

Restrictions may also apply for other reasons. For example, in HTML 4, &#12;, which is a reference to a non-printing "form feed" control character, is allowed because a form feed character is allowed. But in XML, the form feed character cannot be used, not even by reference.[1][citation needed] As another example, &#128;, which is a reference to another control character, is not allowed to be used or referenced in either HTML or XML, but when used in HTML, it is usually not flagged as an error by web browsers – some of which interpret it as a reference to the character represented by code value 128 in the Windows-1252 encoding for compatibility reasons. This character, "€", has to be represented as &#8364; in a standard-compliant HTML code. As a further example, prior to the publication of XML 1.0 Second Edition on October 6, 2000, XML 1.0 was based on an older version of ISO 10646 and prohibited using characters above U+FFFD, except in character data, thus making a reference like &#65536; (U+10000) illegal. In XML 1.1 and newer editions of XML 1.0, such a reference is allowed, because the available character repertoire was explicitly extended.

Markup languages also place restrictions on where character references can occur.

Compatibility issues

[edit]

In the initial versions of SGML and HTML, numeric character references were interpreted in relationship to the document character encoding, rather than Unicode. For Latin-script documents, numeric character references to characters between x80 and x9F in those documents will not be correct against Unicode, and must be recoded. HTML standards prior to HTML 4 supported only Western Latin script documents: the treatment of character references above #7F may vary between applications and national conventions.

For example, as mentioned above, the correct numeric character reference for the Euro sign "€" U+20AC when using Unicode is decimal &#8364; and hexadecimal &#x20AC;. However, if using tools supporting obsolete implementations of HTML, the reference &#128; (Euro sign in the CP-1252 code page) or &#164; (Euro sign in ISO/IEC 8859-15) may work.

As another example, if some text was created originally using the MacRoman character set, the left double quotation mark " will be represented with code point xD2. This will not display properly in a system expecting a document encoded as UTF-8, ISO 8859-1, or CP-1252, where this code point is occupied by the letter Ò. The correct numeric character reference for " in HTML 4 and newer is &#x201C;, because U+201C is its UCS code. In some systems, the named character reference &ldquo; may also be available.

See also

[edit]

References

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Numeric character reference

View on Grokipedia
from Grokipedia
A numeric character reference (NCR) is a markup construct used in SGML and SGML-derived languages such as and XML to represent a specific character by referencing its with a numeric value, allowing the inclusion of characters that may be reserved, difficult to input directly, or unavailable in certain character sets. In , as defined in the Living Standard, an NCR begins with the sequence &# followed by a (optionally prefixed with x for notation) and ends with a (;), such as &#65; for the Latin capital letter A (U+0041) or &#x41; for the same character in hexadecimal form. The converts the numeric value to the corresponding , emitting it as a character token, while handling invalid references—such as those exceeding U+10FFFF or representing surrogate code points—by substituting the replacement character U+FFFD. Semicolons are required for conformance, though their absence triggers a parse error but does not prevent processing in tolerant . Similarly, in XML 1.0 as specified by the W3C, NCRs follow the production CharRef ::= '&#' [0-9]+ ';' for decimal or CharRef ::= '&#x' [0-9a-fA-F]+ ';' for hexadecimal, ensuring representation of legal XML characters within the defined ranges (e.g., U+0020 to U+10FFFF, excluding certain controls). These references must denote valid characters per the XML Char production and are expanded immediately by processors into the referenced character data, facilitating interoperability across diverse encoding environments. Both standards emphasize NCRs as a fundamental mechanism for escaping special characters like < (< or <) and & (& or &) in markup, distinct from named character references that use predefined entity names.

Syntax and Formats

Decimal Form

The decimal form of a numeric character reference begins with an ampersand (&) immediately followed by a number sign (#), then one or more decimal digits (0 through 9) that represent the Unicode code point value in base-10, and ends with a semicolon (;). This form is used to reference Unicode code points in the range from U+0000 to U+10FFFF, subject to the validity rules defined by the markup language specification. Leading zeros are permitted in the decimal sequence and do not change the interpreted value, allowing flexibility in formatting while maintaining equivalence (e.g., &#00065; equals &#65;). For example, &#931; denotes the Greek capital letter sigma (Σ) at code point U+03A3. The decimal form serves as the base-10 alternative to the hexadecimal notation for referencing the same Unicode code points.

Hexadecimal Form

The hexadecimal form of a numeric character reference provides an alternative to the decimal variant by expressing the Unicode code point in base-16 notation. It begins with the sequence &#x (uppercase &#X in HTML only), followed by one or more hexadecimal digits representing the code point value, and terminates with a semicolon (;). This format allows for the inclusion of characters whose code points are more succinctly represented in hexadecimal, particularly for higher values beyond the basic ASCII range. Hexadecimal digits in this reference are case-insensitive, accepting both uppercase (A-F) and lowercase (a-f) letters alongside digits 0-9; for instance, both &#x3A3; and &#x3a3; resolve to the Greek capital letter sigma (Σ, U+03A3). The valid numeric range mirrors that of the decimal form, from U+0000 to U+10FFFF, subject to language-specific restrictions on certain code points such as controls and noncharacters. Leading zeros are permitted but not required, as in &#x000A; for the line feed character (U+000A), though they do not alter the interpreted value. A practical example is &#x2665;, which renders as ♥ (black heart suit, U+2665), demonstrating hexadecimal's advantage in brevity for code points like this one, where the four-digit hex equivalent (2665) is shorter than its five-digit decimal counterpart (9829).

Applications in Markup Languages

In HTML

Numeric character references (NCRs) in HTML serve to represent characters that cannot be directly entered from a keyboard or that might conflict with markup syntax, such as the less-than sign (<) or ampersand (&). For instance, the reserved ampersand can be encoded as &#38; to prevent it from being interpreted as the start of another entity. In HTML5, both decimal and hexadecimal NCR forms are supported, allowing references to any valid Unicode code point from U+0000 to U+10FFFF. The decimal form uses &# followed by decimal digits, while the hexadecimal form uses &#x or &#X followed by hexadecimal digits; both must typically end with a semicolon (;) to terminate the reference. However, semicolons are required except in certain ambiguous cases, such as when the reference is followed by trailing digits that could otherwise extend the numeric value, where parsers may still resolve it for compatibility. NCRs are resolved to their corresponding Unicode characters during parsing, independent of the document's declared encoding, with UTF-8 assumed as the default if none is specified. Invalid NCRs, such as those referencing code points outside the Unicode range or malformed sequences, are typically treated as literal text by parsers or replaced with the Unicode replacement character U+FFFD. HTML5 permits NCRs for C1 control characters (U+0080 to U+009F), which are mapped to specific Unicode equivalents during tokenization in certain contexts, such as attributes, but they may not display consistently across browsers due to varying handling of control codes. A unique aspect of HTML parsing involves resolving ambiguities, such as distinguishing &#59; (encoding a semicolon) from a plain semicolon; most parsers enforce the semicolon terminator to avoid misinterpretation, treating non-terminated sequences as literal ampersands followed by digits. In contrast to XML's stricter requirements, HTML's more lenient approach ensures broader compatibility with legacy content.

In XML and SGML

Numeric character references were introduced in the Standard Generalized Markup Language (SGML), defined by ISO 8879:1986, to reference characters by their numeric position within the document character set, a coded character set specified in the SGML declaration that determines the repertoire of allowable characters. This mechanism allowed authors to include characters not directly available in a limited encoding by substituting them with references like &#N;, where N denotes the character's code position. In SGML, these references are resolved relative to the declared document character set, providing flexibility for various international character sets, though SGML itself does not mandate Unicode. XML, as a subset of SGML, adapted numeric character references to align with Unicode (ISO/IEC 10646), requiring them to denote valid Unicode code points while mandating a terminating semicolon in all cases for unambiguous parsing. Both decimal (&#d;) and hexadecimal (&#xh;) forms have been supported since XML 1.0's initial 1998 recommendation, enabling references to characters across the Unicode range. In XML 1.0 editions prior to 2000, valid code points were restricted to exclude most control characters (e.g., #x1 through #x1F were forbidden both directly and via reference, except for #x9, #xA, and #xD), surrogates (#xD800–#xDFFF), and noncharacters like #xFFFE and #xFFFF, limiting the range effectively to U+0009, U+000A, U+000D, U+0020–U+D7FF, and U+E000–U+FFFD. XML 1.1, introduced in 2004, extended support to the full Unicode repertoire (up to U+10FFFF) by permitting numeric references to additional control characters (#x1–#x1F and #x7F–#x9F, excluding #x0), while maintaining prohibitions on surrogates and requiring special handling in UTF-16 encodings where surrogates might appear in streams. XML parsers enforce numeric character references through strict well-formedness validation, expanding valid ones immediately into character data and treating invalid references—such as those to disallowed code points—as fatal errors that halt processing. This rigorous enforcement contrasts with HTML's more permissive approach, which tolerates certain malformed references for practical web authoring. In both XML 1.0 and 1.1, processors must reject documents containing invalid numeric references to ensure conformance to Unicode semantics and document integrity.

Illustrative Examples

Common ASCII and Latin Characters

Numeric character references (NCRs) provide a standardized way to represent common characters from the ASCII range (Unicode U+0020 to U+007E) and the Latin-1 Supplement (U+0080 to U+00FF), particularly those that are reserved in markup languages or difficult to input on standard keyboards. These references are especially useful for symbols like the ampersand (&) and less-than sign (<), which must be escaped in HTML and XML to avoid interpretation as markup delimiters. In practice, NCRs are frequently employed for the characters & (ampersand), < (less-than), > (greater-than), " (quotation mark), and ' (apostrophe) to prevent entity conflicts and ensure document validity. For the basic ASCII printable characters, NCRs allow direct reference to their Unicode code points. For instance, the ampersand (&, U+0026) is represented as & in decimal form, while the less-than sign (<, U+003C) uses <. Similarly, the greater-than sign (>, U+003E) is >, the quotation mark (", U+0022) is ", and the apostrophe (', U+0027) is '. Letters like uppercase A (A, U+0041) can be denoted as A, though such usage is rare for alphanumeric characters that are easily typed. Extending to the , NCRs facilitate inclusion of accented and symbolic characters common in Western European languages. The (©, U+00A9) is encoded as ©, and the cent sign (¢, U+00A2) as ¢. For ligatures in legacy contexts, such as the small oe (œ, U+0153 in , often mapped from byte 0x9C), the correct NCR is œ in decimal, though older systems might reference it via encoding-specific decimals like 156 for compatibility. These examples highlight how NCRs bridge basic input limitations while adhering to standards.
CharacterDescriptionUnicodeDecimal NCRHexadecimal NCR
&AmpersandU+0026&&
<Less-thanU+003C<<
>Greater-thanU+003E>>
"Quotation markU+0022""
'ApostropheU+0027''
©CopyrightU+00A9©©
¢Cent signU+00A2¢¢
œSmall oe ligatureU+0153œœ

International and Symbolic Characters

Numeric character references (NCRs) facilitate the inclusion of international scripts and symbolic elements in digital documents by allowing direct reference to code points, enabling support for languages and notations outside the basic Latin alphabet. This capability is essential for creating content that spans diverse linguistic traditions and expressive symbols, such as those used in global communication and visual representation. Examples from non-Latin scripts illustrate this utility. The Greek capital letter Alpha (Α), the first letter of the Greek alphabet from the Greek and Coptic block (U+0370–U+03FF), is represented by the decimal NCR Α corresponding to its code point U+0391. Likewise, the Hebrew letter Aleph (א), a foundational consonant in the Hebrew script from the Hebrew block (U+0590–U+05FF), uses the decimal NCR א for U+05D0. For East Asian languages, the CJK Unified Ideograph for "one" (一), the initial character in the CJK Unified Ideographs block (U+4E00–U+9FFF) shared across Chinese, Japanese, and Korean, is denoted by the hexadecimal NCR 一 at U+4E00, highlighting NCRs' role in unifying ideographic systems. Symbolic and emotive characters further demonstrate NCRs' versatility. The black heart suit symbol (♥) from the block (U+2600–U+26FF), often evoking affection, is referenced decimally as ♥ for U+2665. Emojis, residing in higher planes like the Emoticons block (U+1F600–U+1F64F), exemplify extended symbolic use; the grinning face (😀), indicating joy, employs the hexadecimal NCR 😀 for U+1F600. For such elevated code points beyond the Basic Multilingual Plane, hexadecimal notation is typically favored for conciseness over lengthy decimal equivalents, with NCRs specifying the full scalar value directly—distinct from surrogate pair representations in UTF-16 encoding, while handles them as multi-byte sequences.

Historical Development

Origins in SGML

Numeric character references were first defined in the (SGML), formalized as the international standard ISO 8879:1986, where they function as a type of character reference consisting of a delimited character number to specify positions in the reference concrete syntax character set. This reference concrete syntax establishes a standardized mapping for characters in SGML documents, allowing parsers to interpret the numeric values relative to the declared document character set. The core purpose of numeric character references in SGML was to enable the portable representation and interchange of characters across diverse encoding environments, particularly for those not directly supported in the input stream or basic character set, well before the adoption of as a universal standard. By using numeric codes tied to the document's declared character set, SGML documents could maintain consistency and readability regardless of the underlying hardware or software variations in character handling. In early SGML implementations, the syntax for numeric character references primarily employed decimal notation, formatted as &# followed by a sequence of decimal digits and terminated by a semicolon, with these values interpreted relative to the specific document character set rather than a fixed universal coding scheme. Hexadecimal notation, using &#x followed by hexadecimal digits, was also supported but depended on the SGML declaration to define the allowable bases and ranges. A pivotal development occurred with the WebSGML adaptations, incorporated via the second technical corrigendum to ISO 8879 in 1999, which aligned SGML's reference concrete syntax with the standard by mapping the first 65,536 character numbers to Unicode 2.0 s, thereby transitioning numeric references toward universal code point addressing for enhanced web compatibility. This adjustment laid the groundwork for later adaptations in markup languages like and XML.

Evolution in Web Standards

Numeric character references (NCRs), originally derived from SGML as a mechanism for encoding characters by their numeric positions, underwent significant adaptations as web markup languages evolved in the . In , the adoption of NCRs began with the HTML 2.0 specification, published as RFC 1866 in 1995, which introduced basic decimal-form references to represent characters by their code positions in the document character set, aligned with ISO 10646. These decimal NCRs, formatted as &# followed by digits and a , enabled the inclusion of special characters like < (<) within markup, but notation was not yet supported. By HTML 4.01 in 1999, the (W3C) expanded this to include NCRs (&#x followed by hex digits), fully aligning references with code points to facilitate international character support and portability across encodings. Parallel developments occurred in XML, which inherited NCRs from SGML but formalized them in the XML 1.0 Recommendation of February 1998. This initial version supported both and forms from the outset, allowing references to any valid character within the specified repertoire, though limited to exclude certain control and surrogate code points. The second edition of XML 1.0, released in October 2000, refined these rules without altering the core syntax, ensuring consistency with updates to ISO/IEC 10646 and enhancing robustness for broader adoption. A pivotal milestone in the late 1990s was the transition from encoding-dependent interpretations—where NCRs resolved relative to the document's character set—to a Unicode code point-based model, addressing portability challenges in diverse international contexts. This shift, evident in both HTML 4.01 and XML 1.0, laid the groundwork for modern web standards. In the HTML5 era, initiated by the WHATWG in the mid-2000s and formalized as a living standard in 2011, with collaboration between W3C and WHATWG established in 2019 to maintain a unified specification, NCR resolution is standardized to support the full range of Unicode code points (up to U+10FFFF), typically within UTF-8 contexts, ensuring seamless rendering of global scripts and symbols as Unicode evolves (e.g., supporting Unicode 15.0 from 2022 and later versions).

Restrictions and Limitations

Valid Code Points

Numeric character references in markup languages such as and XML are designed to reference code points up to U+10FFFF, though XML excludes U+0000 and other invalid points. This range includes the Basic Multilingual Plane (BMP, U+0000 to U+FFFF), which accommodates the majority of commonly used scripts and symbols, as well as the supplementary planes (U+10000 to U+10FFFF) for less common or historical scripts. For efficiency, hexadecimal notation is typically preferred for code points in the supplementary planes, as their decimal equivalents would involve excessively large numbers. While the syntax permits referencing any code point in this range, best practices emphasize using only assigned code points—those explicitly defined with properties and glyph representations in the Standard—to ensure consistent rendering and behavior across systems. Unassigned code points, which are reserved for potential future assignment, may be technically valid in numeric references but can lead to unpredictable display or interpretation, as their semantics could change in subsequent Unicode versions. As of Unicode 17.0 (released September 9, 2025), 159,801 s are assigned, covering a vast array of languages, symbols, and technical characters. In XML 1.1, the specification explicitly defines valid characters for numeric references as those in the ranges #x1–#xD7FF, #xE000–#xFFFD, and #x10000–#x10FFFF, thereby excluding surrogate code points (U+D800–U+DFFF) and certain noncharacters like U+FFFE and U+FFFF. Although surrogates fall within the overall Unicode range, they are not treated as standalone characters and are recommended to be avoided in encodings like UTF-8, where they form invalid byte sequences. Certain subsets, such as control characters, may face additional restrictions in specific document contexts.

Prohibited and Restricted Characters

Numeric character references (NCRs) in markup languages like XML and are subject to prohibitions and restrictions on certain code points to ensure , , and . These limitations stem from the underlying standard and the specific grammars of XML and , preventing references to code points that could lead to invalid documents or unpredictable behavior in parsers. Control characters, particularly those in the C0 set (U+0000–U+001F) and C1 set (U+0080–U+009F), are often prohibited or restricted in NCRs. In XML 1.0, the valid character production excludes most C0 controls except U+0009 (tab), U+000A (line feed), and U+000D (), rendering NCRs like (form feed, U+000C) invalid as they fall outside the allowed range. XML 1.1 relaxes this slightly by permitting NCRs for controls U+0001–U+001F and certain C1 controls when used in references, though direct inclusion remains forbidden except for whitespace. In , NCRs are barred from referencing U+0000 (null) and most controls beyond ASCII whitespace (U+0009, U+000A, U+000C, U+000D, U+0020), with browsers typically suppressing or ignoring disallowed ones to avoid rendering issues. Surrogate code points (U+D800–U+DFFF) are universally prohibited in NCRs across XML, HTML, and Unicode-compliant encodings like UTF-8 and UTF-16. These ranges are reserved exclusively for surrogate pairs in UTF-16 to encode supplementary characters, and standalone references to them would create malformed sequences incompatible with text interchange. Both XML 1.0/1.1 and explicitly exclude from valid NCR targets to maintain encoding integrity. References to code points beyond the Unicode limit (U+10FFFF) are invalid in all contexts, as they exceed the defined codespace. Noncharacters, such as U+FFFE, U+FFFF, the range U+FDD0–U+FDEF, and the last two positions in each plane (e.g., U+1FFFE–U+1FFFF), are restricted or prohibited; XML 1.0 disallows them entirely in the character production, while HTML permits them technically but treats them as permanently undefined, advising against use to prevent interoperability failures. The private use area (U+E000–U+F8FF) is allowed in NCRs but strongly restricted for practical reasons. Unicode reserves these code points for private agreements without standard semantics, and their use in NCRs can lead to non-interoperable documents, as parsers and applications may interpret them differently or fail to render them consistently; explicitly warns against relying on them for public content.

Compatibility Considerations

Encoding Discrepancies

In pre- HTML documents, numeric character references (NCRs) were interpreted relative to the document's declared , leading to significant discrepancies in character rendering across different charsets. For instance, the NCR � resolves to the euro symbol (€) in documents encoded with (CP-1252), where byte 0x80 maps to U+20AC, but it corresponds to the control character U+0080 (padding character) in strict ISO-8859-1 encoding, where bytes 0x80–0x9F are undefined controls. This mismatch arises because legacy single-byte encodings like extended ISO-8859-1 by assigning printable characters to the 0x80–0x9F range for compatibility with common usage, while ISO-8859-1 reserved them for C1 controls. To reliably represent the euro symbol in Unicode-aware contexts, the correct NCR is € (decimal for U+20AC), which maps consistently regardless of encoding; however, in undeclared or mismatched charsets like without proper declaration, legacy NCRs such as � may fail to render as intended, defaulting to the Unicode control or a replacement character (U+FFFD). Further variations occur with encodings like ISO-8859-15 (Latin-9), which adjusted the ISO-8859-1 layout to include the at byte 0xA4 (NCR ¤), replacing less-used symbols to support the currencies, in contrast to 's placement at 0x80. This positioning difference means ¤ yields a currency sign (¤) in Windows-1252 or ISO-8859-1 but the (€) in ISO-8859-15, highlighting how charset-specific mappings can cause unexpected outputs in cross-encoding scenarios. Modern web standards resolve these issues by mandating UTF-8 as the default and recommended encoding for HTML documents, ensuring NCRs are interpreted directly as Unicode code points (U+0000 to U+10FFFF) independently of the underlying byte sequence. This approach eliminates legacy dependencies, with parsers applying Unicode mappings universally, including the HTML specification's compatibility remapping of certain control NCRs (e.g., � to €) for backward compatibility even in UTF-8 contexts.

Parser and Implementation Variations

Modern web browsers, including Chrome and as of 2025, resolve numeric character references (NCRs) to their corresponding code points during parsing, following the tokenization rules in the living standard. For invalid NCRs, such as those exceeding the range (e.g., � referencing a code point beyond U+10FFFF), browsers emit the replacement character U+FFFD instead of the literal text, ensuring graceful degradation without halting parsing. XML parsers, such as , enforce strict adherence to the XML 1.0 specification, requiring a terminating for all NCRs and raising a if absent (e.g., &#x3C treated as invalid without the ;). In contrast, HTML parsers like html5lib adopt a more tolerant approach aligned with recovery, inferring the end of an NCR by consuming available digits even without a and emitting the resolved character if valid, while flagging it as a . Certain edge cases highlight implementation differences: some older tools, particularly those emulating early XML or processing in strict modes, disallow hexadecimal NCRs altogether, treating them as unrecognized syntax rather than resolving to equivalents. Additionally, NCRs for characters in the astral planes (e.g., U+1F000 for ) render inconsistently or fail entirely on legacy systems lacking font support for supplementary planes, often displaying as blank boxes or fallback glyphs instead of the intended symbols. The HTML living standard, maintained as a continuously updated document through 2025, mandates consistent surrogate pair handling for astral plane characters resolved via NCRs, ensuring that code points from U+10000 to U+10FFFF are properly represented in UTF-16 without erroneous surrogate emission.

References

  1. https://html.spec.whatwg.org/multipage/parsing.html#character-references
  2. https://www.w3.org/TR/xml/#NT-CharRef
  3. https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state
  4. https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference
  5. https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-state
  6. https://html.spec.whatwg.org/multipage/syntax.html#hexadecimal-character-references
  7. https://www.w3.org/TR/xml/#sec-character-references
  8. https://www.w3.org/TR/xml/#NT-Char
  9. https://html.spec.whatwg.org/multipage/syntax.html#character-references
  10. https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-state
  11. https://html.spec.whatwg.org/multipage/syntax.html#syntax-character-references-0
  12. https://html.spec.whatwg.org/multipage/parsing.html#parse-error-control-character-reference
  13. https://html.spec.whatwg.org/multipage/parsing.html#c1-controls
  14. https://html.spec.whatwg.org/multipage/parsing.html#ambiguous-ampersand
  15. https://www.w3.org/MarkUp/html-spec/charset-harmful
  16. https://www.w3.org/TR/NOTE-sgml-xml
  17. https://www.w3.org/TR/xml/
  18. https://www.w3.org/TR/1998/REC-xml-19980210
  19. https://www.w3.org/TR/2006/REC-xml11-20060816/
  20. https://www.w3.org/TR/html401/charset.html
  21. https://www.unicode.org/charts/PDF/U0300.pdf
  22. https://www.unicode.org/charts/PDF/U0590.pdf
  23. https://www.unicode.org/charts/PDF/U4E00.pdf
  24. https://www.unicode.org/charts/PDF/U2600.pdf
  25. https://www.unicode.org/charts/PDF/U1F600.pdf
  26. https://www.iso.org/obp/ui/#iso:std:iso:8879:ed-1:en
  27. https://www.loc.gov/preservation/digital/formats/fdd/fdd000465.shtml
  28. https://sgmljs.sgml.net/docs/sgmlrefman.html
  29. https://openjade.sourceforge.net/doc/sgmldecl.htm
  30. https://www.w3.org/TR/html401/charset.html#h-5.3
  31. https://www.w3.org/TR/1998/REC-xml-19980210#sec-references
  32. https://www.w3.org/TR/2000/REC-xml-20001006
  33. https://www.w3.org/blog/2019/w3c-and-whatwg-to-work-together-to-advance-the-open-web-platform/
  34. https://html.spec.whatwg.org/multipage/parsing.html
  35. https://www.w3.org/TR/xml11/#NT-Char
  36. https://www.unicode.org/versions/Unicode17.0.0/
  37. https://qntm.org/safe
  38. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/
  39. https://infra.spec.whatwg.org/#control
  40. https://infra.spec.whatwg.org/#surrogate
  41. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-2/
  42. https://infra.spec.whatwg.org/#noncharacter
  43. https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state
  44. http://www.i18nguy.com/markup/ncrs.html
  45. https://www.i18nqa.com/debug/table-iso8859-1-vs-iso8859-15.html
  46. https://www.w3.org/International/questions/qa-choosing-encodings
  47. https://stackoverflow.com/questions/49588726/emoji-unicode-does-not-render-into-emoji
Add your contribution
Related Hubs
Contribute something
User Avatar
No comments yet.