Overview
Once work crosses a process or network boundary, synchronous call-and-wait stops scaling and starts failing in new ways: partial failure, reordering, duplication. This pattern language — messaging, event-driven architecture, sagas, idempotency — exists to make asynchrony reliable. The fallacies of distributed computing (the network is not reliable, latency is not zero…) are the reason every one of these patterns exists.
Key points
- Messaging styles: point-to-point queues (work distribution) vs. pub/sub topics (fan-out); message brokers (RabbitMQ) vs. event logs (Kafka — replayable, ordered per partition).
- Delivery guarantees: at-most-once, at-least-once, "exactly-once" (in practice: at-least-once + idempotent consumers). Idempotency is the cornerstone pattern — design every consumer and API to tolerate duplicates.
- Event-driven architecture: event notification vs. event-carried state transfer vs. event sourcing — three very different commitments often conflated.
- Data + messaging consistency: transactional outbox (write event and state in one transaction, relay asynchronously), change data capture, listen-to-yourself.
- Distributed workflows: sagas (choreography vs. orchestration) replace distributed transactions; compensating actions instead of rollback; 2PC largely avoided.
- Resilience patterns: timeouts, retries with exponential backoff + jitter, circuit breakers, bulkheads, dead-letter queues, backpressure (reactive streams).
- Ordering & time: no global clock — happens-before, logical/vector clocks, and why "the latest write" is a design decision, not a fact.
- To explore: consensus (Raft), exactly-once in Kafka (transactions), CQRS, workflow engines (Temporal) as durable execution.
Practice
- Gossip Glomers: Echo & Unique IDs (source) — warm up with the Maelstrom harness, then generate globally unique IDs without coordination.
- Gossip Glomers: Broadcast (source) — gossip a value to every node through injected partitions: retries, at-least-once delivery, and idempotent handlers in the flesh.
- Transactional outbox relay (source) — implement the pattern against a real database: state + event in one transaction, an async relay, and a consumer that provably tolerates the duplicates.
- Build a message broker (source) — Crickett's NATS challenge: pub/sub and queue groups from the wire protocol up, where delivery guarantees stop being abstract.
- Implement Raft with Gossip Glomers (source) — the escape hatch when order must be total: build the consensus these patterns exist to avoid paying for.
Related
- System design fundamentals
- Microservice architecture — the context where most of these patterns become mandatory.
- Concurrency and Parallelism — the single-machine counterpart of the same problems.
- Cache management — event-driven invalidation.
- Message brokers & event streaming — the infrastructure deep dive (Kafka internals) behind these patterns.