Monedula

Scenario 01.3.3 · Idempotent dedup vs duplicate

Kafka idempotent producer dedup vs a plain producer

Producers & EOS · Intermediate · about 3 min · Apache Kafka 4.3

A lost ack: an idempotent producer is deduped where a plain producer writes a duplicate - and a dropped batch trips OUT_OF_ORDER_SEQUENCE_NUMBER.

Run this scenario in the playground

Dropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops). t+11.100s

A batch the broker never received left a gap after the last accepted sequence, so the broker rejects the next batch with OUT_OF_ORDER_SEQUENCE_NUMBER. The simulator stops the producer here, as a transactional producer would have to; a Kafka 4.x idempotent producer instead bumps its epoch, resets the sequence and retries on its own.

Open at this step: step 6, Dropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops)

What you start with

Two producers face the same lost ack side by side. The idempotent producer stamps each record with a per-partition sequence number, so the broker recognises the retried record and dedups it; the plain producer has no sequence and writes a visible duplicate. A dropped batch then leaves a sequence gap that the broker rejects as OUT_OF_ORDER_SEQUENCE_NUMBER.

Step by step

  1. Two producers on orders/0 - idempotence ON (p-idem) vs OFF (p-plain) t+0.000s Open at this step: step 1, Two producers on orders/0 - idempotence ON (p-idem) vs OFF (p-plain)
  2. p-idem commits X at offset 0 (producerSeq 0) t+1.100s The broker records last-accepted sequence 0 for p-idem on this partition. Open at this step: step 2, p-idem commits X at offset 0 (producerSeq 0)
  3. Lost ack → p-idem re-sends seq 0 → DEDUPED (no second write) t+3.100s Idempotence: the broker recognises the already-committed sequence and acks without appending. Still one record. Open at this step: step 3, Lost ack → p-idem re-sends seq 0 → DEDUPED (no second write)
  4. p-plain commits X at offset 1 t+5.100s Open at this step: step 4, p-plain commits X at offset 1
  5. Lost ack → p-plain re-sends → DUPLICATE at offset 2 t+7.100s No idempotence: the retry writes a second copy. The same record now sits at offsets 1 and 2. Open at this step: step 5, Lost ack → p-plain re-sends → DUPLICATE at offset 2
  6. Dropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops) t+11.100s A batch the broker never received left a gap after the last accepted sequence, so the broker rejects the next batch with OUT_OF_ORDER_SEQUENCE_NUMBER. The simulator stops the producer here, as a transactional producer would have to; a Kafka 4.x idempotent producer instead bumps its epoch, resets the sequence and retries on its own. Open at this step: step 6, Dropped batch left a gap → p-idem hits OUT_OF_ORDER_SEQUENCE_NUMBER (producer stops)

The takeaway

Idempotence gives exactly-once-per-partition writes: the broker's per-partition sequence numbers dedup retries that a plain producer would duplicate, and detect gaps from a lost batch instead of silently misordering the log.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.3.3" title="Kafka Simulator: Idempotent dedup vs duplicate" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/idempotence-dedup/">Kafka idempotent producer dedup vs a plain producer</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

More in Producers & EOS

All scenarios