rgoussu.dev
rgoussu@goussu: ~/library/fundamentals
❯ ~/library/fundamentals cat unicode-and-text-encoding.md

Unicode & text encoding

# Code points, UTF-8, normalization, and grapheme clusters — why "length" lies and how text really works.

Conceptsaved 2026-08-08updated 2026-08-09 #unicode#encoding#utf-8#text#fundamentals

Overview

Text is the most-handled data type in software and the least-understood: "string" hides a three-layer tower — bytes (an encoding), code points (Unicode's numbers), and grapheme clusters (what a human calls a character) — and most text bugs are confusion between layers. A few hours of deliberate study permanently cures mojibake, broken emoji slicing, "é ≠ é" comparison failures, and the annual UTF-8 incident.

Key points

  • The three layers: bytes ←(encoding)→ code points ←(segmentation)→ grapheme clusters. é can be one code point (U+00E9) or two (e + combining accent); 👩‍👩‍👧 is one grapheme, several code points, many bytes. Every "length" API answers for exactly one layer — know which: Go's len counts bytes, Java/JS .length counts UTF-16 code units (surrogate pairs count double), Rust makes you choose, Python counts code points — none of them counts what users see.
  • UTF-8 won for reasons: ASCII-compatible, self-synchronizing (you can find the next character boundary from anywhere), no byte-order issues, compact for ASCII-heavy text. It is the interchange default: declare it, assume nothing — mojibake is always a write-with-one, read-with-another mismatch, and "there is no plain text" (an undeclared encoding is a guess).
  • The UTF-16 legacy tax: Java, JavaScript, C#, and Windows APIs speak UTF-16 internally — surrogate pairs are where astral characters (emoji, many CJK) break naive code; slicing a string mid-surrogate produces garbage. Iterate by code point (or grapheme, with a library/ICU) — never by unit index arithmetic.
  • Normalization: the same visible text has multiple encodings — NFC (composed, the storage/interchange default) vs. NFD (decomposed); compare and deduplicate after normalizing or "é" ≠ "é" bugs ensue (classic in filenames: macOS historically stored NFD). NFKC folds compatibility characters (fi → fi) — useful for identifiers/search, lossy for display.
  • Case and comparison are locale-bound: uppercase of i is İ in Turkish — the famous bug family; case-insensitive comparison means case folding, and sorting means collation (locale-specific, what databases collations are about). Never lowercase security-relevant identifiers with locale-default functions.
  • Security edges: homoglyph spoofing (Cyrillic а vs. Latin a — IDN/identifier attacks), normalization-bypass of validation (validate after normalizing — AppSec input-handling rule applied to text), invisible characters (zero-width, bidi controls — the "Trojan Source" trick).
  • To explore: the encoding zoo you still meet (Latin-1, Windows-1252, Shift-JIS — and charset detection's limits), Punycode, ICU as the reference library, how emoji ZWJ sequences work.

Practice

  • "The Absolute Minimum About Unicode" (source) — Spolsky's canonical read-first; the bytes/code-points mental model in one sitting.
  • UTF-8 codec + text CLI (exercise) — encode and decode UTF-8 by hand from RFC 3629, then build a small text-processing tool on top — the three-layer tower made concrete.
  • Build Your Own JSON parser (source) — handle \uXXXX escapes and surrogate pairs correctly; where astral characters go wrong in practice.
  • Grapheme segmentation drill (source) — implement (or exercise a library's) grapheme iteration and prove naive .length mis-slices an emoji ZWJ sequence.

Related