PRACTICAL CHARACTER GUIDE
Unicode normalization and accented letters
Two accented letters can look identical on screen while using different code-point sequences underneath. Unicode normalization gives software a consistent way to handle those equivalent forms.
01
One visible letter can have more than one valid sequence
The letter é can be encoded as the single precomposed character U+00E9. It can also be represented by U+0065, LATIN SMALL LETTER E, followed by U+0301, COMBINING ACUTE ACCENT. Unicode defines these sequences as canonically equivalent.
A font often renders both forms as the same glyph. That visual agreement does not make their bytes or code-point counts identical. Exact comparisons, cursor movement, validation, database constraints, filenames, and search pipelines can expose the difference.
02
What NFC and NFD do
NFD performs canonical decomposition, separating characters into their canonical components where defined. NFC first decomposes canonically and then composes eligible sequences. For common Latin text, NFC often turns a base letter plus combining mark into a precomposed character.
Neither form is more linguistically correct. They are standardized representations for software interchange. Some operating systems and data sources favor different forms, which is why text can change sequence after moving between a form, file, API, and database.
- NFC: canonical decomposition followed by composition
- NFD: canonical decomposition without recomposition
- Both preserve canonical meaning
- Normalization does not choose the correct spelling or accent
03
NFKC and NFKD have a broader effect
NFKC and NFKD apply compatibility decomposition in addition to canonical decomposition. Compatibility mappings can fold presentation or formatting distinctions into a common underlying form. This is useful in selected search and identifier workflows, but it can remove distinctions an application intended to preserve.
Do not use compatibility normalization as a reflexive cleanup step. Define the comparison goal, review the character classes in the actual data, and retain original input when fidelity matters. Security-sensitive identifiers need a separate, explicit policy beyond ordinary accent normalization.
04
Where normalization problems appear
A form can accept an accented name, store it in decomposed form, and later compare it with a precomposed account record. A search index can tokenize one representation differently from another. A filename copied between systems can look unchanged while failing an exact lookup.
JavaScript string length counts UTF-16 code units rather than visible grapheme clusters, so it is not a reliable count of what a person sees. Even code-point counting does not always equal grapheme counting because a displayed character can contain several code points.
05
A practical implementation policy
For ordinary web text, use UTF-8 and choose NFC as a consistent application boundary unless a platform protocol requires something else. Normalize before equality checks and search indexing when the product specification allows it. Keep the unmodified source value when exact evidence or round-trip fidelity is important.
Apply normalization at known boundaries rather than repeatedly throughout the codebase. Test precomposed and decomposed fixtures, stacked Vietnamese marks, letters without precomposed forms, emoji sequences, and scripts beyond Latin. Normalization solves equivalence, not every Unicode text problem.
- Document the chosen form
- Normalize both sides of a comparison
- Test multilingual fixtures
- Keep original input when exact preservation matters
- Never use normalization to strip accents from names
06
How to inspect a suspicious character
Paste one user-perceived character into the character identifier to compare its entered, NFC, and NFD code points. If two strings still behave differently after the same normalization, inspect whitespace, punctuation, invisible controls, letter lookalikes, and application-specific transformations.
A rendering problem is usually separate. If the code points are correct but a box appears, the font may lack a glyph. If text becomes é, the bytes were probably decoded with the wrong character encoding rather than the wrong normalization form.
Continue with the right reference
Sources: Unicode Standard Annex #15: Unicode Normalization Forms, Unicode normalization FAQ.