Kafka producer config
enable.idempotence: the idempotent producer
enable.idempotence makes the Kafka producer write each record exactly once per partition even when it retries: the broker recognises a retried batch by its producer id and sequence number and drops the copy. It is on by default since Kafka 3.0, and needs acks=all, retries above 0 and max.in.flight.requests.per.connection of 5 or less.
- Config
enable.(producer)idempotence - Default
- true (since Kafka 3.0)
- Valid values
- true, false
What it does
A producer that gets no answer to a request cannot tell whether the write failed or only the acknowledgement was lost, so it retries. Without idempotence, a retry of a write that did succeed appends a second copy. With idempotence, the producer gets a producer id from the broker and numbers each batch per partition; the broker remembers the last sequence numbers it accepted, acknowledges a duplicate without appending it, and rejects a batch that arrives out of order, which also keeps order with up to 5 requests in flight.
The trade-off
Idempotence costs almost nothing, which is why it is the default. Its limits are scope: it only covers retries by the same producer instance within one session, not resends by application code or a producer that restarts and replays its input. For those, use transactions with a transactional.id, or make the consumer idempotent. If acks, retries or max.in.flight.requests.per.connection conflict with it and enable.idempotence was not set explicitly, the producer quietly turns it off.
Advice
- Set enable.idempotence=true explicitly, so that a conflicting setting fails with a ConfigException instead of turning idempotence off silently.
- Keep acks=all and leave retries and delivery.timeout.ms at their defaults.
- For read-process-write pipelines, add transactions and read_committed consumers on top; idempotence alone does not cover them.
See it in the simulator
- Kafka idempotent producer dedup vs a plain producer (Idempotent dedup vs duplicate) Step 3: Lost ack → p-idem re-sends seq 0 → DEDUPED (no second write) Open at this step: Kafka idempotent producer dedup vs a plain producer
- Kafka enable.idempotence=false: a retry writes a duplicate (The duplicate problem) Step 4: Duplicate: O1 now at offsets 0 AND 1 Open at this step: Kafka enable.idempotence=false: a retry writes a duplicate
- Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS (Idempotence retry) Step 4: broker-2 back - the in-flight retry commits exactly once Open at this step: Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS