Messaging in Go — Kafka, NATS & AMQP — Library

Overview

Messaging in Go is all community libraries, and for Kafka specifically it is the one protocol with four credible clients rather than one winner — the choice is a real decision. NATS is the counterweight: written in Go, culturally Go-native, and often the default async fabric in all-Go shops where the JVM world would reflexively reach for Kafka or RabbitMQ. The cross-cutting concerns — delivery semantics, schema management, testing against real brokers — matter more than the client choice and transfer across all of them.

Key points

franz-go (twmb/franz-go): the modern full-featured Kafka client — pure Go, supports every KIP that matters (transactions, exactly-once, incremental rebalancing), high performance; the current default for new serious Kafka work.
segmentio/kafka-go: the idiomatic one — Reader/Writer types that feel like Go I/O, easy to pick up; fewer advanced features than franz-go.
sarama (Shopify/sarama, now IBM/sarama): the legacy incumbent — everywhere in existing codebases, API shows its age (consumer groups are clunky); maintained, but new projects rarely start here.
confluent-kafka-go: cgo bindings over librdkafka — feature-complete and Confluent-supported, but cgo costs you trivial cross-compilation and static binaries, which is exactly what Go teams value.
NATS (nats-io/nats.go): core NATS for fire-and-forget pub/sub and request-reply (sync RPC over the bus, a pattern Kafka can't do naturally); JetStream adds persistence, at-least-once delivery, consumer acks, KV and object stores — Kafka-ish durability with radically simpler ops (one small binary).
RabbitMQ: rabbitmq/amqp091-go (successor to streadway/amqp) — channels, exchanges, manual acks; you hand-roll reconnect logic or wrap it (wagslane/go-rabbitmq).
watermill (ThreeDotsLabs): the abstraction layer — one Publisher/Subscriber interface with pub/sub implementations for Kafka, NATS, AMQP, SQL, and router middleware (retry, poison queue, correlation); Go's rough analogue to Spring Cloud Stream.
Delivery semantics: design for at-least-once everywhere — idempotent consumers (dedup on a message ID or natural key), and the outbox pattern for atomically publishing alongside DB writes; exactly-once exists in Kafka transactions but don't build correctness on it across system boundaries.
Schema management: protobuf or Avro + a schema registry (Confluent SR — clients via franz-go's sr package or srclient; buf's registry for proto); enforce backward-compatible evolution in CI, never by convention.

Details

Kafka client comparison

Client	Implementation	Feature depth	Feel	Pick when
franz-go	Pure Go	Full (EOS, transactions, KIPs)	Powerful, denser API	New services, advanced Kafka features
segmentio/kafka-go	Pure Go	Core	Idiomatic Reader/Writer	Straightforward produce/consume
IBM/sarama	Pure Go	Broad but dated	Verbose	Existing codebases already on it
confluent-kafka-go	cgo (librdkafka)	Full, vendor-backed	C-ish	Confluent support contract; librdkafka parity required

NATS: the road less travelled by the JVM

Core NATS gives subjects with wildcards, queue groups for load balancing, and request-reply with microsecond latencies — an RPC-and-events fabric in one. JetStream layers streams (persisted subjects), durable consumers with explicit ack/nak/term, and retention policies. The operational pitch: a single ~20MB binary clusters trivially, versus Kafka's brokers + controllers + partition maths. The trade-off: a smaller ecosystem for heavyweight stream processing — no Kafka Streams equivalent; long-horizon replay and massive-throughput event sourcing still favour Kafka.

Testing against real brokers

Fakes lie about ordering, rebalancing and redelivery — test against the real thing with testcontainers-go: modules/kafka (or modules/redpanda, faster to boot and wire-compatible), modules/rabbitmq, and a plain container for nats. Spin the broker per test package, produce/consume through your actual code, and assert on redelivery by killing the consumer mid-batch.

Examples

// franz-go: consume with a group, commit after processing (at-least-once).
cl, _ := kgo.NewClient(
    kgo.SeedBrokers("localhost:9092"),
    kgo.ConsumerGroup("billing"),
    kgo.ConsumeTopics("orders"),
    kgo.DisableAutoCommit(),
)
for {
    fetches := cl.PollFetches(ctx)
    fetches.EachRecord(func(r *kgo.Record) { process(r) }) // must be idempotent
    cl.CommitUncommittedOffsets(ctx)
}

// NATS request-reply: sync RPC over the bus.
nc, _ := nats.Connect(nats.DefaultURL)
msg, err := nc.Request("orders.get", []byte(id), 2*time.Second)

Backend protocols in Go — the map — parent map; where async fits among the protocols.
Message brokers and event streaming — broker-side architecture these clients talk to.
API documentation in Go — documenting these channels: AsyncAPI's thin Go story and the schema-registry boundary.
Integration testing in Go — testcontainers-go patterns for broker tests.
Messaging in Java — Kafka, JMS & AMQP — the JVM contrast: spec-shaped (JMS) and framework-wrapped where Go picks a client library; NATS conspicuously absent there.

Messaging in Go — Kafka, NATS & AMQP