Scenario 04.1.1 · At-most-once, at-least-once, exactly-once

Kafka at-most-once vs at-least-once vs exactly-once

Transactions · Intermediate · about 3 min · Apache Kafka 4.3

Three producers, three contracts on one partition: a lost ack duplicates under at-least-once, dedups under exactly-once, and acks=0 silently drops a record on a dying leader.

What you start with

Kafka's three message-delivery contracts side by side on one partition: at-least-once (acks=all, idempotence off), exactly-once (acks=all, idempotence on), and at-most-once (acks=0, retries=0). Each producer writes its own key, and a single lost ack - plus a leader killed mid-write - reveals how each contract reacts to the same fault.

Step by step

1. Three producers, three contracts t+0.300s Three producers share one partition of 'orders', each wired for a different delivery contract: p-alo (acks=all, idempotence off) is at-least-once, p-eos (acks=all, idempotence on) is exactly-once, and p-amo (acks=0) is at-most-once. The log is empty. Open at this step: step 1, Three producers, three contracts
2. ALO committed (acks=all) t+1.300s p-alo produces ALO with acks=all; the leader appends it at offset 0 and the followers replicate it, so the high watermark advances to 1. The record is durably committed exactly once - so far. Open at this step: step 2, ALO committed (acks=all)
3. Lost ack → at-least-once duplicate t+2.300s p-alo's ack was dropped, so the producer retried. With idempotence OFF the broker cannot tell the resend from a new record and appends ALO again at offset 1 - the log now holds ALO twice (LEO/HW 2). This is the at-least-once contract: no loss, but duplicates are possible. Open at this step: step 3, Lost ack → at-least-once duplicate
4. EOS committed (acks=all, idempotent) t+3.300s p-eos produces EOS with acks=all AND idempotence ON; it commits at offset 2 (HW 3). Every idempotent record carries a producer id + sequence number that the broker remembers. Open at this step: step 4, EOS committed (acks=all, idempotent)
5. Lost ack → exactly-once dedup t+4.300s p-eos's ack was dropped and it retried too - but the resend carries the same producer sequence, so the broker recognises it as a duplicate and discards it. The LEO stays at 3: EOS is stored exactly once. That is the exactly-once contract (per partition). Open at this step: step 5, Lost ack → exactly-once dedup
6. AMO1 fire-and-forget, replicated t+5.500s p-amo (acks=0) sends AMO1 and never waits for an ack - its status goes straight to 'sent'. This one happens to replicate before anything fails, committing at offset 3 (HW 4). acks=0 gives you no guarantee that it will. Open at this step: step 6, AMO1 fire-and-forget, replicated
7. AMO2 in flight (acks=0, 'sent') t+5.750s p-amo sends AMO2 and immediately reports 'sent' without waiting. AMO2 is on the leader's log (broker-1 LEO 5) but the followers haven't fetched it yet, so it is NOT committed - the high watermark is still 4. Open at this step: step 7, AMO2 in flight (acks=0, 'sent')
8. Leader dies → AMO2 silently lost t+6.300s broker-1 (the leader) is killed before AMO2 replicated. broker-2 is elected leader at LEO 4 - it never received AMO2, so the record is gone and the producer was never told. That is at-most-once: deliver zero or one time, never redelivered. Final committed log: ALO twice, EOS once, AMO1 once - AMO2 lost. Open at this step: step 8, Leader dies → AMO2 silently lost

The takeaway

The identical fault yields three different logs: at-least-once duplicates, exactly-once dedups, and at-most-once silently drops - your acks and idempotence settings decide which guarantee you actually get.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
• retries How many times the producer sends a batch again after a transient error (default 2147483647, so in practice delivery.timeout.ms ends the retrying); idempotence needs it above 0.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=04.1.1" title="Kafka Simulator: At-most-once, at-least-once, exactly-once" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/delivery-contracts/">Kafka at-most-once vs at-least-once vs exactly-once</a></p>

Further reading

Apache Kafka documentation

More in Transactions

• At-least-once delivery in Kafka: where duplicates come from
• Kafka transaction commit vs abort with read_committed
• Kafka isolation.level: read_committed vs read_uncommitted
• Kafka last stable offset (LSO) with concurrent transactions
• Kafka transaction.timeout.ms: the coordinator aborts
• Kafka read_committed during a leader election
• Kafka consume-process-produce loop with exactly-once
• Kafka zombie producer fencing with transactional.id
• Kafka processing.guarantee=exactly_once_v2 end to end

All scenarios