rgoussu.dev
rgoussu@goussu: ~/library/fundamentals/exercises
❯ ~/library/fundamentals/exercises cat utf8-codec-and-text-cli-subject.md

UTF-8 codec + text-processing CLI — subject

# The work statement for a hand-rolled UTF-8 encoder/decoder from RFC 3629 plus a layer-aware text CLI — byte-pattern rules, the invalid sequences to reject, and test cases from é to the family emoji.

Subjectsaved 2026-08-08source #unicode#utf-8#encoding#text#fundamentals#exercise#subject

Brief

"Length" lies, emoji slice in half, and files arrive as mojibake — all because text is three different things stacked: bytes, code points, and grapheme clusters. You are going to earn the distinction by implementing UTF-8 yourself, straight from RFC 3629, with no help from your language's built-in codec, and then build a small wc-like CLI that counts a string three ways and shows why the numbers disagree. Do the bit-shifting by hand once and mojibake stops being mysterious.

Instructions

Encoder — code point → bytes (RFC 3629)

Map a Unicode scalar value to 1–4 bytes by range:

  • U+0000 – U+007F → 1 byte: 0xxxxxxx (the code point as-is).
  • U+0080 – U+07FF → 2 bytes: 110xxxxx 10xxxxxx.
  • U+0800 – U+FFFF → 3 bytes: 1110xxxx 10xxxxxx 10xxxxxx.
  • U+10000 – U+10FFFF → 4 bytes: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx.

The x bits carry the code point's bits, most-significant first; each continuation byte holds 6 bits. Reject surrogates U+D800–U+DFFF (never valid scalar values) and anything above U+10FFFF. Always emit the shortest form for the code point (no overlong encodings).

Decoder — bytes → code points (reject the invalid)

Read a byte stream back to code points. The lead byte's high bits give the length; each following byte must be 10xxxxxx. Reject, with a clear error (or a documented replacement policy), every malformed case:

  • Overlong encodings — a code point encoded in more bytes than necessary (e.g. C0 80 for NUL). Enforce the minimum code point per length (≥ 0x80 for 2-byte, ≥ 0x800 for 3-byte, ≥ 0x10000 for 4-byte).
  • Surrogate code points U+D800–U+DFFF appearing in the stream.
  • Stray/lone continuation bytes (10xxxxxx with no lead) and invalid lead bytes (0xC0, 0xC1, 0xF5–0xFF).
  • Truncated sequences — a multi-byte sequence cut short by end-of-input or a non- continuation byte.

Verify the codec against your language's built-in UTF-8 on a fixtures file (round-trip valid input; agree on which invalid inputs are rejected).

Self-synchronization demo

From a random byte offset into a UTF-8 buffer, find the next character boundary using only the high bits — a boundary is any byte that is not a continuation byte (10xxxxxx). Show this recovers cleanly mid-stream, which UTF-16 cannot do.

The layer-aware CLI

Build a small wc-like tool that, on the same input, reports three counts:

  • bytes — raw length,
  • code points — decoded scalar values (your decoder),
  • grapheme clusters — user-perceived characters (extended grapheme cluster segmentation).

Make the three numbers visibly differ on a combining-accent é (e + U+0301: 3 bytes, 2 code points, 1 grapheme) and on a ZWJ emoji sequence. Add a validate subcommand (report the first invalid byte offset and why) and a transcode subcommand between Latin-1 and UTF-8.

Normalization pass

Add an option to NFC-normalize input, and demonstrate that two visually identical strings (precomposed é U+00E9 vs. decomposed e+U+0301) compare unequal as bytes but equal after NFC.

Examples

Test cases the tool must handle correctly:

Input bytes code points graphemes note
A 1 1 1 ASCII
é (U+00E9, precomposed) 2 1 1 2-byte encode
é (e + U+0301) 3 2 1 combining mark
家 (U+5BB6) 3 1 1 3-byte (CJK)
👨‍👩‍👧‍👦 (family, ZWJ) 25 7 1 ZWJ cluster
\x80 (lone continuation) — reject — invalid: stray continuation
\xC0\x80 (overlong NUL) — reject — invalid: overlong
\xED\xA0\x80 (surrogate) — reject — invalid: surrogate
\xE2\x28 — reject — invalid: truncated/bad continuation

(Byte counts assume UTF-8; the family emoji is 4 people-emoji joined by 3 ZWJs = 7 code points, one grapheme.)

Constraints

  • Do not use the language's built-in UTF-8 encode/decode for the codec itself — implement the bit layout by hand. The built-in may only be used as an oracle to check your results and for grapheme segmentation / NFC where a Unicode-data library is reasonable.
  • Follow RFC 3629 strictly: 4 bytes max, U+10FFFF ceiling, no surrogates, shortest form only.
  • Invalid input must be detected, not silently mangled — reject or apply a documented replacement policy.

Acceptance

  • Milestone 1 (encoder): every code point in U+0000–U+10FFFF (minus surrogates) encodes to the correct 1–4 bytes; surrogates and out-of-range values are rejected. Matches the reference encoder on the fixtures.
  • Milestone 2 (decoder): valid streams decode to the right code points; overlongs, lone continuation bytes, surrogates, and truncated sequences are all rejected. Agrees with the language codec on the fixtures file.
  • Milestone 3 (self-sync): from an arbitrary offset the tool finds the next boundary using only the high bits and resumes decoding correctly.
  • Milestone 4 (layer-aware CLI): the three counts are correct and visibly differ for the combining-é and ZWJ-emoji cases per the table; validate and Latin-1↔UTF-8 transcode work.
  • Milestone 5 (normalization): the NFC option makes precomposed vs. decomposed é compare equal after normalization though their bytes differ.

Related