Scenario 04.1.2 · Where duplicates come from
At-least-once delivery in Kafka: where duplicates come from
A lost ack - not a lost record - makes an acks=all producer retry, and with producer idempotence off the broker appends the same order twice.
Run this scenario in the playgroundOne acks=all producer, idempotence off. t+0.300s
A single producer on the at-least-once contract: acks=all (so writes are durable) with enable.idempotence=false set explicitly, since the default has been true since Kafka 3.0. This is the exact configuration that turns a retry into a duplicate.
Open at this step: step 1, One acks=all producer, idempotence offORDER-42 appended and replicated. t+1.400s
The producer sends ORDER-42. The leader appends it at offset 0, the ISR replicates it, and the broker sends the ack. HW=1 - the record is safely stored.
Open at this step: step 2, ORDER-42 appended and replicatedAck lost → producer retries. t+1.700s
The ack never reaches the producer. From the producer's side a lost ack is indistinguishable from a lost write, so - exactly as its contract requires - it resends ORDER-42. The leader (broker-1, LEO=2) already has the resend ahead of the followers.
Open at this step: step 3, Ack lost → producer retriesRetry appends ORDER-42 again → duplicate. t+2.600s
With idempotence off the resend carries no producer id and sequence, so the broker cannot tell it from a brand-new record and appends ORDER-42 again at offset 1. The log now holds the same order twice and every consumer reads it twice. THIS is where at-least-once duplicates come from: a retry after an uncertain outcome. Turning on idempotence (see 01.3.x) lets the broker recognise the resend and dedup it.
Open at this step: step 4, Retry appends ORDER-42 again → duplicateWhat you start with
A close-up on the birth of a single at-least-once duplicate. One acks=all producer with idempotence OFF writes ORDER-42; the broker stores and replicates it, but the acknowledgement is lost in the network, so the producer - unable to tell a lost write from a lost ack - retries.
Step by step
- One acks=all producer, idempotence off t+0.300s A single producer on the at-least-once contract: acks=all (so writes are durable) with enable.idempotence=false set explicitly, since the default has been true since Kafka 3.0. This is the exact configuration that turns a retry into a duplicate. Open at this step: step 1, One acks=all producer, idempotence off
- ORDER-42 appended and replicated t+1.400s The producer sends ORDER-42. The leader appends it at offset 0, the ISR replicates it, and the broker sends the ack. HW=1 - the record is safely stored. Open at this step: step 2, ORDER-42 appended and replicated
- Ack lost → producer retries t+1.700s The ack never reaches the producer. From the producer's side a lost ack is indistinguishable from a lost write, so - exactly as its contract requires - it resends ORDER-42. The leader (broker-1, LEO=2) already has the resend ahead of the followers. Open at this step: step 3, Ack lost → producer retries
- Retry appends ORDER-42 again → duplicate t+2.600s With idempotence off the resend carries no producer id and sequence, so the broker cannot tell it from a brand-new record and appends ORDER-42 again at offset 1. The log now holds the same order twice and every consumer reads it twice. THIS is where at-least-once duplicates come from: a retry after an uncertain outcome. Turning on idempotence (see 01.3.x) lets the broker recognise the resend and dedup it. Open at this step: step 4, Retry appends ORDER-42 again → duplicate
The takeaway
Duplicates come from retries after an uncertain outcome, not from lost data; a broker receiving no producer id and sequence can't recognise the resend and appends the order twice - enabling idempotence is what closes the gap.
Configs in this scenario
-
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works -
enable.Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off. How enable.idempotence worksidempotence
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=04.1.2" title="Kafka Simulator: Where duplicates come from" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/where-duplicates-come-from/">At-least-once delivery in Kafka: where duplicates come from</a></p>Further reading
Troubleshooting
- Kafka duplicate messages: causes and fixes
- Why does a Kafka consumer reprocess messages after restart?
More in Transactions
- Kafka at-most-once vs at-least-once vs exactly-once
- Kafka transaction commit vs abort with read_committed
- Kafka isolation.level: read_committed vs read_uncommitted
- Kafka last stable offset (LSO) with concurrent transactions
- Kafka transaction.timeout.ms: the coordinator aborts
- Kafka read_committed during a leader election
- Kafka consume-process-produce loop with exactly-once
- Kafka zombie producer fencing with transactional.id
- Kafka processing.guarantee=exactly_once_v2 end to end