---
title: "At-least-once delivery in Kafka: where duplicates come from"
description: "ORDER-42 is stored and replicated, but the ack is lost. Unable to tell a lost ack from a lost write, the producer retries and Kafka appends the order twice."
url: https://monedula.dev/kafka-simulator/scenarios/where-duplicates-come-from/
---

# At-least-once delivery in Kafka: where duplicates come from

Scenario 04.1.2 · Where duplicates come from. Transactions · Intermediate · about 3 min · Apache Kafka 4.3.

A lost ack - not a lost record - makes an acks=all producer retry, and with producer idempotence off the broker appends the same order twice.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2

## What you start with

A close-up on the birth of a single at-least-once duplicate. One acks=all producer with idempotence OFF writes ORDER-42; the broker stores and replicates it, but the acknowledgement is lost in the network, so the producer - unable to tell a lost write from a lost ack - retries.

## Step by step

1. **One acks=all producer, idempotence off** (t+0.300s): A single producer on the at-least-once contract: acks=all (so writes are durable) with enable.idempotence=false set explicitly, since the default has been true since Kafka 3.0. This is the exact configuration that turns a retry into a duplicate. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2&t=0)
2. **ORDER-42 appended and replicated** (t+1.400s): The producer sends ORDER-42. The leader appends it at offset 0, the ISR replicates it, and the broker sends the ack. HW=1 - the record is safely stored. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2&t=1)
3. **Ack lost → producer retries** (t+1.700s): The ack never reaches the producer. From the producer's side a lost ack is indistinguishable from a lost write, so - exactly as its contract requires - it resends ORDER-42. The leader (broker-1, LEO=2) already has the resend ahead of the followers. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2&t=2)
4. **Retry appends ORDER-42 again → duplicate** (t+2.600s): With idempotence off the resend carries no producer id and sequence, so the broker cannot tell it from a brand-new record and appends ORDER-42 again at offset 1. The log now holds the same order twice and every consumer reads it twice. THIS is where at-least-once duplicates come from: a retry after an uncertain outcome. Turning on idempotence (see 01.3.x) lets the broker recognise the resend and dedup it. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2&t=3)

## The takeaway

Duplicates come from retries after an uncertain outcome, not from lost data; a broker receiving no producer id and sequence can't recognise the resend and appends the order twice - enabling idempotence is what closes the gap.

## Configs in this scenario

- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)
- [`enable.idempotence`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_enable.idempotence): Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off. [How enable.idempotence works](https://monedula.dev/kafka-simulator/configs/enable-idempotence/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#message-delivery-semantics)

## Troubleshooting

- [Kafka duplicate messages: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/duplicate-messages/)
- [Why does a Kafka consumer reprocess messages after restart?](https://monedula.dev/kafka-simulator/troubleshooting/consumer-reprocesses-messages-after-restart/)