rgoussu.dev
rgoussu@goussu: ~/library/system-design
❯ ~/library/system-design cat message-brokers-and-event-streaming.md

Message brokers & event streaming

# Kafka internals and the broker landscape — logs vs. queues, partitions, consumer groups, replication, and exactly-once semantics.

Conceptsaved 2026-08-08 #kafka#messaging#event-streaming#distributed-systems#system-design

Overview

The deep dive behind the async patterns: how brokers actually work, with Kafka as the centerpiece because its design — a partitioned, replicated, append-only log — has become the reference model for event streaming. Understanding the internals (partitions, consumer groups, replication protocol, delivery semantics) is what turns "we use Kafka" into correctly reasoning about ordering, throughput, and failure.

Key points

  • Log vs. queue is the fundamental split: queues (RabbitMQ, SQS) delete on ack — work distribution; logs (Kafka, Pulsar, Redpanda) retain and let consumers replay from any offset — events as durable history, multiple independent readers.
  • Kafka's anatomy: topics → partitions (the unit of ordering and parallelism; keys route to partitions — ordering holds per key only), segments on disk, sequential I/O + zero-copy + batching as the reason it's fast.
  • Consumer groups: one partition ↔ one consumer per group; rebalancing (and its pauses); offset commits are where at-least-once vs. at-most-once is actually decided; consumer lag as the health metric.
  • Replication: leader/follower per partition, in-sync replicas (ISR), acks=all + min.insync.replicas as the durability contract; KRaft replacing ZooKeeper for metadata consensus.
  • Delivery semantics: idempotent producer (dedup per partition), transactions across partitions, and "exactly-once" as exactly-once stream processing (read-process-write atomically) — not magic end-to-end delivery; consumers still need idempotency at the edges.
  • The ecosystem: Kafka Connect (CDC in, sinks out), stream processing (Kafka Streams, Flink), schema registry — Avro/Protobuf with enforced compatibility rules as the contract layer for events.
  • To explore: partitioning strategy & hot-key skew, retention vs. compaction, tiered storage, when RabbitMQ or NATS is honestly the better fit.

Practice

  • Gossip Glomers: Kafka-Style Log (source) — implement a replicated log with offsets and committed reads: the append-only model in miniature, checked by Maelstrom.
  • Build a message broker (NATS) (source) — Crickett's challenge: pub/sub from the wire protocol up — the queue side of the log-vs-queue split.
  • Build your own Redis — pub/sub stretch (source) — add SUBSCRIBE/PUBLISH to the Redis build: fan-out with no retention, the exact opposite corner from Kafka's durable history.
  • Build your own Kafka (source) — CodeCrafters' staged build: speak the Kafka wire protocol, then add partitions and consumer offsets; the anatomy above stops being folklore.

Related