rgoussu.dev
rgoussu@goussu: ~/library/fundamentals/exercises
❯ ~/library/fundamentals/exercises cat utf8-codec-and-text-cli.md

UTF-8 codec + text-processing CLI

# Encode and decode UTF-8 by hand from the spec, then build a small text CLI on top — the bytes/code-points/graphemes tower turned into working code.

Exercisesaved 2026-08-08source #unicode#utf-8#encoding#text#fundamentals#exercise

Goal

Implement a UTF-8 encoder and decoder from RFC 3629 without leaning on the language's built-in codec, then build a small text-processing CLI on top of it. Doing the bit-shifting yourself makes the three-layer tower — bytes, code points, grapheme clusters — permanent, and cures mojibake, broken emoji slicing, and "length lies" for good.

Subject: full brief & instructions

Practices

  • Unicode & text encoding — the three layers, UTF-8's self-synchronizing design, normalization, and the "length" traps this drills.
  • AppSec fundamentals — validate after normalizing; homoglyph and invisible-character handling are input-trust rules applied to text.

Milestones

  1. Encoder. Map a code point to 1–4 bytes per RFC 3629: compute the length from the code-point range, lay out the leading-byte and continuation-byte bit patterns. Reject surrogates (U+D800–DFFF) and out-of-range values.
  2. Decoder. Read a byte stream back to code points; reject overlong encodings, stray continuation bytes, and truncated sequences — the invalid inputs are the whole point. Verify against your language's codec on a fixtures file.
  3. Self-synchronization demo. From a random byte offset, find the next character boundary using only the high bits — proving why UTF-8 recovers where UTF-16 can't.
  4. Text CLI, layer-aware. Build a small wc-like tool that reports counts three ways — bytes, code points, and grapheme clusters — on the same input; make the numbers differ on an emoji ZWJ sequence and a combining-accent é.
  5. Normalization pass. Add an option to NFC-normalize input and show that two visually identical strings compare equal only after normalization.

Stretch goals

  • Detect and flag bidi-control / zero-width characters (the "Trojan Source" trick).
  • Add a UTF-16 (surrogate-pair) codec and cross-convert.

Related