Scenario 01.3.2 · The duplicate problem

Kafka enable.idempotence=false: a retry writes a duplicate

Producers & EOS · Intermediate · about 2 min · Apache Kafka 4.3

With idempotence OFF (acks=all), a lost ack makes the producer retry and append a SECOND copy of the same record - a visible duplicate in the log.

What you start with

A record can be committed durably yet have its ack lost on the way back to the producer. With idempotence OFF there is no per-partition sequence number to dedup against, so when the producer retries the broker appends a second copy. Watch O1 commit at offset 0, then reappear at offset 1 after drop_ack forces a retry.

Step by step

1. Idle · producer ready (idempotence OFF) t+0.000s Three in-sync replicas, acks=all, min.ISR=2. The producer has enable.idempotence=false, so it attaches no producer id or per-partition sequence number for the broker to dedup retries against. Open at this step: step 1, Idle · producer ready (idempotence OFF)
2. O1 committed at offset 0 t+1.500s O1 is produced and replicated to the ISR; the high watermark advances to 1. The producer received its ack - the write is durable. Open at this step: step 2, O1 committed at offset 0
3. Ack lost - producer retries t+3.000s inject_produce_fault drop_ack cancels the in-flight ack. The producer never hears that O1 committed, so it retries: the leader re-appends O1 at offset 1 (leader LEO=2) and the producer waits for the second ack while the copy replicates (HW still 1). Open at this step: step 3, Ack lost - producer retries
4. Duplicate: O1 now at offsets 0 AND 1 t+4.000s The re-sent copy replicates to the ISR and the high watermark advances to 2. With no sequence number to recognise the re-send, the broker treats it as a brand-new record - the log holds O1 twice. Retries without idempotence cause duplicates. Open at this step: step 4, Duplicate: O1 now at offsets 0 AND 1

The takeaway

Without idempotence, at-least-once delivery means a lost ack produces a real duplicate in the log - the exact problem the idempotent producer was designed to eliminate.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• enable.idempotence Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off. How enable.idempotence works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.3.2" title="Kafka Simulator: The duplicate problem" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/the-duplicate-problem/">Kafka enable.idempotence=false: a retry writes a duplicate</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• Kafka duplicate messages: causes and fixes

More in Producers & EOS

• Kafka producer acks=0: fire and forget, silent data loss
• Kafka producer acks=1: acknowledged by the leader only
• Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR
• Kafka acks=1 data loss: min.insync.replicas does not help
• Kafka linger.ms vs batch.size: when the producer sends
• Kafka producer partitioners: keyed, round-robin and sticky
• Kafka compression.type: fewer, larger producer batches
• Kafka buffer.memory and max.block.ms: producer backpressure
• Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS
• Kafka idempotent producer dedup vs a plain producer
• Kafka max.in.flight.requests.per.connection and ordering

All scenarios