---
title: "Kafka enable.idempotence=false: a retry writes a duplicate"
description: "An acks=all record commits but its ack is lost. With idempotence off the Kafka producer retries and the broker appends a second copy of the same record."
url: https://monedula.dev/kafka-simulator/scenarios/the-duplicate-problem/
---

# Kafka enable.idempotence=false: a retry writes a duplicate

Scenario 01.3.2 · The duplicate problem. Producers & EOS · Intermediate · about 2 min · Apache Kafka 4.3.

With idempotence OFF (acks=all), a lost ack makes the producer retry and append a SECOND copy of the same record - a visible duplicate in the log.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=01.3.2

## What you start with

A record can be committed durably yet have its ack lost on the way back to the producer. With idempotence OFF there is no per-partition sequence number to dedup against, so when the producer retries the broker appends a second copy. Watch O1 commit at offset 0, then reappear at offset 1 after drop_ack forces a retry.

## Step by step

1. **Idle · producer ready (idempotence OFF)** (t+0.000s): Three in-sync replicas, acks=all, min.ISR=2. The producer has enable.idempotence=false, so it attaches no producer id or per-partition sequence number for the broker to dedup retries against. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=01.3.2&t=0)
2. **O1 committed at offset 0** (t+1.500s): O1 is produced and replicated to the ISR; the high watermark advances to 1. The producer received its ack - the write is durable. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=01.3.2&t=1)
3. **Ack lost - producer retries** (t+3.000s): inject_produce_fault drop_ack cancels the in-flight ack. The producer never hears that O1 committed, so it retries: the leader re-appends O1 at offset 1 (leader LEO=2) and the producer waits for the second ack while the copy replicates (HW still 1). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=01.3.2&t=2)
4. **Duplicate: O1 now at offsets 0 AND 1** (t+4.000s): The re-sent copy replicates to the ISR and the high watermark advances to 2. With no sequence number to recognise the re-send, the broker treats it as a brand-new record - the log holds O1 twice. Retries without idempotence cause duplicates. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=01.3.2&t=3)

## The takeaway

Without idempotence, at-least-once delivery means a lost ack produces a real duplicate in the log - the exact problem the idempotent producer was designed to eliminate.

## Configs in this scenario

- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)
- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`enable.idempotence`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_enable.idempotence): Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off. [How enable.idempotence works](https://monedula.dev/kafka-simulator/configs/enable-idempotence/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#message-delivery-semantics)

## Troubleshooting

- [Kafka duplicate messages: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/duplicate-messages/)