rgoussu.dev
rgoussu@goussu: ~/library/system-design
❯ ~/library/system-design cat rate-limiting-and-idempotency.md

Rate limiting & idempotency

# The two guardrails of a public API — bounding request rates without unfairness, and making retries safe by design.

Conceptsaved 2026-08-08 #rate-limiting#idempotency#api#resilience#system-design

Overview

These two patterns pair naturally because together they make an API safe to call from an unreliable world: rate limiting protects the server from too many requests, idempotency protects the client's data from retried ones. Both look trivial ("count requests", "dedupe by key") and both hide real distributed-systems problems the moment you run more than one node.

Key points

  • Rate-limiting algorithms: token bucket (allows bursts up to capacity; the usual default), leaky bucket (smooths to constant rate), fixed window (cheap; edge-of-window double-burst flaw), sliding window log/counter (accurate; costlier). Know the burst behavior of each — it's the actual differentiator.
  • Distributed limiting: per-node limits multiply by node count; central counters (Redis + Lua for atomicity) buy accuracy with latency; the pragmatic middle is local buckets refilled from a central allowance. Exactness is rarely worth the coordination — approximately-right and fast wins.
  • Limit by what, and answer well: key on user/API-key first (IP as fallback, NAT-polluted); tier limits per endpoint cost; respond 429 with Retry-After and rate-limit headers — a limit clients can't observe produces retry storms, the very thing it exists to prevent. Distinguish throttling (delay) from rejection, and load-shedding (server-health-driven) from rate limiting (client-fairness-driven).
  • Idempotency, precisely: an operation whose repetition leaves state as if executed once. HTTP's GET/PUT/DELETE are idempotent by contract; POST is not — hence idempotency keys: client sends a unique key, server stores key → result and replays the stored result on retry (Stripe's model).
  • The subtleties: the key-check and the operation must be atomic (unique constraint in the same transaction — a check-then-act race defeats the purpose); store the response, not just "seen"; scope keys per operation and expire them; concurrent duplicates get 409/wait, not double execution.
  • Idempotency is the retry license: only idempotent operations may be retried blindly — which is why at-least-once delivery (async patterns) makes consumer idempotency non-negotiable, and why exponential backoff + jitter belongs on every retry path that holds one.
  • To explore: adaptive/concurrency limits (AIMD, gradient), quota systems vs. rate limits, exactly-once semantics as idempotency wearing a suit.

Practice

  • Rate limiter algorithms (source) — Crickett's challenge: token bucket, leaky bucket, fixed and sliding windows behind one interface; unit-test each one's burst behavior at the window edge — the actual differentiator.
  • Stripe-style idempotency keys (source) — middleware storing key → response with the check and the operation atomic; race two concurrent duplicates to prove the unique constraint earns its keep.
  • Build a load balancer & rate limiter (source) — the limiter grown up: per-key limits at an edge tier, then Redis-backed distributed counters — approximately-right-and-fast vs. exact, measured.

Related