Build a load balancer & rate limiter

Goal

Build the two guardrails of an API edge — a reverse-proxying load balancer and a rate limiter (both are John Crickett Coding Challenges) — then compose them into one small edge tier. Proves the boxes at the border of every architecture diagram are things you can build, break, and measure, not just draw.

Subject: full brief & instructions

Practices

• System design fundamentals — L4/L7 load balancing, health checks, stateless scaling.
• Rate limiting & idempotency — the four algorithms and their burst behavior; distributed counters vs. local buckets.
• API design — 429, Retry-After, and rate-limit headers as observable contract surface.
• Performance engineering — measuring the edge's own overhead without lying to yourself.

Milestones

1. Reverse proxy — forward HTTP to a single backend, relaying headers and status faithfully; diff responses direct vs. proxied to prove transparency.
2. Round robin — a static backend list with rotation; a dead backend produces a 502, not a crash.
3. Health checks — periodic checks eject an unhealthy backend and re-admit it on recovery; kill a backend under live traffic and watch requests reroute.
4. Rate limiter, in process — token bucket and fixed window behind one interface; unit-test the edge-of-window double burst fixed window allows and token bucket doesn't.
5. Sliding window — add sliding window log and counter; chart what each algorithm lets through under the same bursty client.
6. Compose the edge — per-API-key limiting inside the balancer: 429 + Retry-After + rate-limit headers, so a well-behaved client can back off.
7. Distributed counters — run two balancer instances sharing Redis-backed counters (atomic via Lua); measure accuracy vs. latency against per-node local buckets.
8. Load test — drive it with wrk2 through the edge vs. direct to the backend; report the p99 overhead your edge costs.

Stretch goals

• Least-connections and weighted balancing strategies.
• Graceful drain: finish in-flight requests on shutdown before removing a node.
• WebSocket passthrough — Upgrade support, generous idle timeouts, sticky routing (WebSockets & bidirectional protocols).
• Adaptive concurrency limits (AIMD) instead of fixed quotas.

Related

• Build a load balancer & rate limiter — subject — the self-contained work statement: LB and limiter specs, composition, distributed counters, and the load-test protocol.
• System design fundamentals — cites this exercise from its Practice list.
• Rate limiting & idempotency — likewise; milestones 4–7 are its algorithms made concrete.
• API design — likewise, for the response-surface milestones.
• Microservice architecture — likewise; this is the gateway tier every service system needs.
• WebSockets & bidirectional protocols — likewise, via the WebSocket-passthrough stretch.
• Build your own Redis — kin exercise; it can even supply the counter store for milestone 7.