Messaging in Java — Kafka, JMS & AMQP

Overview

Messaging decouples services in time: a producer writes to a broker and moves on; a consumer processes when ready. The Java landscape splits into three families — Kafka (a partitioned, replayable event log with its own protocol), JMS/Jakarta Messaging (the Java-standard API over brokers like ActiveMQ Artemis), and AMQP (RabbitMQ's wire protocol with rich routing). Each framework wraps them differently, but the hard problems are broker-independent: delivery semantics, idempotency, atomic write-plus-publish, and schema evolution on the wire.

Key points

• kafka-clients is the foundation: KafkaProducer/KafkaConsumer, consumer groups, partition assignment, offset commits — every framework integration wraps it.
• Spring Kafka: @KafkaListener methods and KafkaTemplate, container-managed concurrency, error handlers with retries and a DeadLetterPublishingRecoverer.
• Spring Cloud Stream abstracts the broker behind binders (Kafka, Rabbit): you write Function<In, Out>/Consumer<In> beans bound to channels by configuration — portable, at the cost of distance from broker-specific features.
• Quarkus uses SmallRye Reactive Messaging (MicroProfile Reactive Messaging): @Incoming/@Outgoing channel methods, Mutiny streams, connectors for Kafka, AMQP and JMS; Emitter bridges imperative code into channels.
• Micronaut Kafka: compile-time @KafkaListener consumers and declarative @KafkaClient producer interfaces; sibling modules cover JMS and RabbitMQ.
• JMS / Jakarta Messaging (jakarta.jms): queues and topics behind a standard API; ActiveMQ Artemis is the reference broker; Spring gives JmsTemplate and @JmsListener.
• AMQP / RabbitMQ: exchanges, bindings and routing keys; Spring AMQP provides RabbitTemplate and @RabbitListener.
• Delivery is at-least-once by default everywhere sensible — design consumers to be idempotent instead of chasing exactly-once.
• Serialization: JSON for convenience; Avro or protobuf with a schema registry (Confluent, Apicurio) for contracts that must evolve safely.
• Test against real brokers with Testcontainers, not in-memory imitations.

Details

Kafka integrations compared

Kafka Streams, in one paragraph: a client-side processing library (no cluster of its own) that turns topics into KStream/KTable topologies — map, join, window, aggregate — with local state stores backed by changelog topics and exactly-once processing within the Kafka domain. Spring Boot and Quarkus both configure a topology bean; reach for it when consumers stop being "handle one record" and become continuous stream transformation.

JMS and AMQP

JMS is an API standard, not a wire protocol: code against jakarta.jms and swap brokers. Artemis (the modern ActiveMQ engine) is the usual choice; Quarkus connects via the SmallRye JMS/AMQP connectors, Micronaut via micronaut-jms. RabbitMQ's AMQP 0-9-1 model adds broker-side routing — producers publish to exchanges, bindings route to queues by routing key — which makes fan-out, topic routing and per-consumer queues first-class. Rule of thumb: Kafka for event streams you may replay and for throughput; Rabbit/Artemis for classic work queues, request/reply and routing-heavy integration.

Delivery semantics honestly stated

• At-most-once: commit/ack before processing — you lose messages on crash. Rarely what you want.
• At-least-once: process, then commit — the default posture; crash between the two redelivers, so consumers must be idempotent (natural keys, upserts, dedup tables).
• "Exactly-once" is scoped, not universal: Kafka's idempotent producer plus transactions give exactly-once within Kafka (consume-process-produce, Kafka Streams EOS); the moment a side effect leaves Kafka (a DB write, an email) you are back to at-least-once plus idempotency.
• Outbox pattern solves atomic "update DB and publish": write the event to an outbox table in the business transaction, relay it to the broker afterwards (Debezium CDC or a poller) — see the message-brokers note for the full treatment.
• Ordering holds per Kafka partition (choose keys accordingly) and per queue elsewhere; DLQs need monitoring, or they are silent data loss with extra steps.

Serialization and schema evolution

JSON is fine between two services you own; at organisational scale use Avro (or protobuf) with a schema registry — producers register schemas, consumers resolve them by id from the record, and the registry enforces compatibility modes (backward/forward/full) so a producer cannot silently break its consumers. Apicurio is the open registry commonly paired with Quarkus; Confluent's with Spring. Treat the schema as API and review it like one.

Testing

Testcontainers modules for Kafka, RabbitMQ and Artemis spin the real broker per test class — the only way serializer config, rebalancing and transaction behaviour get exercised honestly. Spring's @EmbeddedKafka is faster but not the real broker; keep it for narrow listener-wiring tests. Awaitility handles the asynchrony (await().untilAsserted(...)) — never Thread.sleep.

Examples

// Spring Kafka
@KafkaListener(topics = "orders", groupId = "billing")
void on(Order order) { ... }

// Quarkus — SmallRye Reactive Messaging
@Incoming("orders")
@Outgoing("invoices")
Invoice process(Order order) { ... }

Related

• Backend protocols in Java — the map — parent overview of the protocol landscape.
• Message brokers and event streaming — the broker-side concepts (logs vs queues, outbox, CDC) this note implements in Java.
• API documentation in Java — documenting these channels: AsyncAPI, Springwolf and the schema-registry boundary.
• Integration testing — Testcontainers practice for broker-backed tests.
• Java frameworks — the Spring/Quarkus/Micronaut context these integrations live in.
• gRPC in Java — the synchronous counterpart for service-to-service calls.