rgoussu.dev
rgoussu@goussu: ~/library/system-design
❯ ~/library/system-design cat cache-management.md

Cache management

# Caching strategies, invalidation, eviction, and the failure modes — from CPU lines to CDNs and distributed caches.

Conceptsaved 2026-08-08 #caching#system-design#performance#distributed-systems

Overview

Caching trades freshness and memory for latency and load: keep a copy of expensive-to-get data closer to the consumer. It appears at every layer — CPU, application memory, distributed stores (Redis, Memcached), HTTP/CDN — and the hard part is never storing the data but keeping it correct: "there are only two hard things in computer science: cache invalidation and naming things."

Key points

  • Read strategies: cache-aside (app manages the cache; most common), read-through (cache loads on miss); write strategies: write-through (synchronous, consistent, slower), write-behind (fast, risks loss), write-around.
  • Invalidation: TTL as the baseline, explicit invalidation on write, event-driven invalidation via CDC or pub/sub — pick per tolerance for staleness.
  • Eviction policies: LRU, LFU, FIFO, and modern variants (TinyLFU, ARC, Caffeine's W-TinyLFU) — the policy determines the hit rate under your access pattern.
  • Failure modes to design for: cache stampede / thundering herd (mitigate with request coalescing, locks, probabilistic early expiry), cache penetration (negative caching, bloom filters), hot keys (replication, key splitting), cold start after flush.
  • Distributed caching: consistent hashing for placement, local + remote two-tier setups, the consistency gap between cache and source of truth.
  • HTTP caching is a protocol: Cache-Control, ETag/If-None-Match, stale-while- revalidate — the cheapest cache is the one in the client or CDN.
  • To explore: cache coherence in CPUs (MESI) as the low-level mirror of the same problem, materialized views as "caches with a schema".

Practice

  • LRU cache (source) — the classic kata: O(1) get/put from a hash map + doubly linked list, the mechanics under every eviction discussion.
  • LFU cache (source) — the harder sibling; maintaining frequency buckets shows why real systems settle for approximations like TinyLFU.
  • Build your own Memcached (source) — Crickett's challenge: a networked cache speaking the memcached text protocol, with TTLs and concurrent clients.
  • Build your own Redis (exercise) — the full build: RESP protocol, passive + active expiry, eviction under a memory cap, and a benchmark to prove it.
  • Tokio mini-redis (exercise) — the same server async in Rust: shared state across tasks, framing, and cancellation-by-drop.

Related