Systems at scale — distributed systems & data

Goal

Architect-level system design is pattern-matching over things you have actually felt break. This path builds that inventory: how data is stored, cached, moved, and agreed upon across machines — with the KB's heaviest build projects as the proof at each step. The outcome is the ability to design a system under real constraints and defend every box on the diagram, which is also precisely the system-design-interview skill.

Prerequisites

• Foundations in full; Language craft modules 3 or 4 (the builds here want Go or Rust in hand).

Modules

1. The trade-off vocabulary

• Read: System design fundamentals.
• Do: the note's drills — design three classic systems on paper (URL shortener, feed, chat) using the vocabulary precisely.
• Proof: each design names its consistency, partitioning, and replication choices and what was traded away.

2. Storage from the inside

• Read: Databases and other storage systems.
• Do: Build your own database.
• Proof: the subject's gates — B-tree splits correct, WAL replay survives a mid-write kill, the secondary index pays for itself in a measured query.

3. Caching, priced

• Read: Cache management.
• Do: Build your own Redis.
• Proof: the subject's benchmark protocol met; expiry and eviction semantics correct under the checker; a written stance on when a cache is the wrong fix.

4. Moving data — async patterns & brokers

• Read: Asynchronous and distributed system patterns and Message brokers & event streaming.
• Do: their practice drills — an outbox + idempotent consumer pair on a real service; a partitioned consumer-group lab where rebalancing is observed, not read about.
• Proof: a duplicate-delivery test that your consumer survives; the delivery-guarantee ladder explained with your own lab as evidence.

5. The public edge

• Read: Rate limiting & idempotency and WebSockets & bidirectional protocols.
• Do: Build a load balancer & rate limiter.
• Proof: the subject's load-test protocol — fair limiting under bursts, correct 429/Retry-After surface, health-checked balancing.

6. Agreement

• Read: Distributed consensus.
• Do: Implement Raft with Gossip Glomers.
• Proof: the Maelstrom workloads green through the Raft rung; you can explain what consensus refuses to solve and where you'd buy it instead of building it.

7. Decomposition, soberly

• Read: Microservice architecture and Search systems.
• Do: their drills — decompose a monolith you know on paper along its actual fracture planes; stand up an inverted-index lab next to a database and route queries honestly.
• Proof: the decomposition names its distributed-systems bill (from modules 4–6) service by service.

8. Performance engineering & the data platform

• Read: Performance engineering and Data engineering.
• Do: End-to-end ELT pipeline; profile one of this path's earlier builds under load and fix the top bottleneck.
• Proof: the pipeline's CI gates green with a backfill demonstrated; the optimization documented measurement-first like the 1BRC ladder.

Related

• The path to excellence — the master path; Stage 4's core.
• Platform & production — running what this path designs.
• Language craft — supplied the network-server muscle these builds assume.
• Architect & lead — turns this inventory into design leadership.
• Themes drawn on: System Design, Data Storage.