Scenario 01.3.1 · Idempotence retry

Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS

Producers & EOS · Intermediate · about 2 min · Apache Kafka 4.3

An idempotent acks=all produce retries through NotEnoughReplicas and commits exactly once.

What you start with

With two brokers down the ISR is below min.insync.replicas, so an acks=all produce is rejected with NOT_ENOUGH_REPLICAS. NOT_ENOUGH_REPLICAS is retriable, so the producer retries with backoff within delivery.timeout.ms, and idempotence keeps those retries free of duplicates; once a broker restarts and the ISR recovers the write commits exactly once.

Step by step

1. Cluster ready - acks=all, idempotence on t+0.000s Open at this step: step 1, Cluster ready - acks=all, idempotence on
2. Two brokers down - ISR shrinks to 1 (below min.ISR) t+5.100s Open at this step: step 2, Two brokers down - ISR shrinks to 1 (below min.ISR)
3. Produce hits NotEnoughReplicas - retrying t+7.100s NOT_ENOUGH_REPLICAS is retriable, so the producer keeps retrying; idempotence makes each retry safe. Open at this step: step 3, Produce hits NotEnoughReplicas - retrying
4. broker-2 back - the in-flight retry commits exactly once t+8.300s No duplicate: the same record lands at offset 0 once the cluster heals. Open at this step: step 4, broker-2 back - the in-flight retry commits exactly once
5. Stable - record committed at offset 0 t+10.000s Open at this step: step 5, Stable - record committed at offset 0

The takeaway

An idempotent acks=all producer turns a transient NOT_ENOUGH_REPLICAS into a safe retry: the record commits once the ISR heals, with no loss and no duplicate.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• delivery.timeout.ms The upper bound on reporting success or failure after send() returns, covering the wait in a batch, the wait for the acknowledgement and every retry (default 120000, 2 minutes); it should be at least request.timeout.ms plus linger.ms.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.3.1" title="Kafka Simulator: Idempotence retry" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/idempotence-retry/">Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• NotEnoughReplicasException in Kafka: causes and fixes

More in Producers & EOS

• Kafka producer acks=0: fire and forget, silent data loss
• Kafka producer acks=1: acknowledged by the leader only
• Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR
• Kafka acks=1 data loss: min.insync.replicas does not help
• Kafka linger.ms vs batch.size: when the producer sends
• Kafka producer partitioners: keyed, round-robin and sticky
• Kafka compression.type: fewer, larger producer batches
• Kafka buffer.memory and max.block.ms: producer backpressure
• Kafka enable.idempotence=false: a retry writes a duplicate
• Kafka idempotent producer dedup vs a plain producer
• Kafka max.in.flight.requests.per.connection and ordering

All scenarios