Kafka troubleshooting

Consumers see duplicate messages

Kafka symptom

Duplicates come from one of two places. Either the record is in the log twice, because a producer retried or resent it without idempotence, or it is in the log once and the consumer processed it twice, because it restarted or rebalanced before committing its offset. Check the offsets: the same payload at two offsets is a producer duplicate, the same offset processed twice is a consumer redelivery.

Reproduce it in the playground

Scenario 01.3.2 · The duplicate problem Seen in 2 more scenarios

The moment it happensDuplicate: O1 now at offsets 0 AND 1. t+4.000s

The re-sent copy replicates to the ISR and the high watermark advances to 2. With no sequence number to recognise the re-send, the broker treats it as a brand-new record - the log holds O1 twice. Retries without idempotence cause duplicates.

Open at this step: step 4, Duplicate: O1 now at offsets 0 AND 1

What it means

Kafka's default contract is at least once. A producer that sends a record and gets no response cannot tell whether the write failed or only the acknowledgement was lost, so it retries, and logs each retry as a warning such as "Got error produce response with correlation id 12 on topic-partition orders-0, retrying (2147483646 attempts left). Error: REQUEST_TIMED_OUT. Error Message: Disconnected from node 1 due to timeout". Without idempotence, a retry of a batch the broker had already written is a duplicate. The idempotent producer stamps every batch with a producer id and a sequence number, so the broker discards a retried batch it already has, and since Kafka 3.0 enable.idempotence defaults to true together with acks=all. On the consumer side, the committed offset is where a group resumes, so records processed after the last commit are delivered again after a crash or when the partition moves to another member. Neither side gives exactly-once processing on its own: that takes transactions for Kafka-to-Kafka pipelines, or an idempotent consumer when the output goes to another system.

Likely causes, most common first

Idempotence is off. enable.idempotence=false turns it off, and so does a conflicting setting such as acks=1 or max.in.flight.requests.per.connection above 5, silently, unless idempotence was enabled explicitly. A retry after a lost acknowledgement then appends a second copy of the record.
The application resends. Idempotence only covers the producer's own retries within one producer session. Code that catches a send failure and calls send() again, or a producer process that restarts and replays its input, writes new records the broker has no way to recognise as repeats.
The consumer is redelivered records it already processed. With auto commit (enable.auto.commit=true, every auto.commit.interval.ms, 5 seconds by default) or with manual commits after each batch, a crash, a kill or a rebalance between processing and the next commit makes the next owner start from the last committed offset and process those records again.
A read_uncommitted consumer reads aborted transactions. isolation.level defaults to read_uncommitted, which also returns records from aborted transactions. If a transactional producer aborts and retries, such a consumer sees both attempts.

How to fix it

Leave enable.idempotence at its default, or set it to true explicitly so that a conflicting setting raises a ConfigException instead of disabling it without a word. Keep acks=all.
Do not resend from application code. Let the producer retry within delivery.timeout.ms, and treat a failure reported after that as an incident to investigate rather than a reason to send again.
For consume, transform and produce pipelines, use transactions: a transactional.id on the producer, the consumed offsets committed with sendOffsetsToTransaction, and isolation.level=read_committed on every downstream consumer.
For writes to databases and other external systems, make the consumer idempotent: upsert by a business key, or store the processed offset next to the data in the same transaction.
Commit processed offsets in ConsumerRebalanceListener.onPartitionsRevoked, so a rebalance does not hand uncommitted work to the next owner.

What not to do

Committing offsets before processing to stop redelivery. That is at-most-once: after a crash, the records between the commit and the crash are never processed at all.
Turning retries off or lowering acks to stop producer duplicates. Transient errors then reach the application as failed sends, and acks below all gives up durability; idempotence removes the duplicates without either cost.

Configs involved

enable.idempotence Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off.
acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
max.in.flight.requests.per.connection How many unacknowledged requests the producer sends on one connection (default 5); above 1 without idempotence a retried batch can land after a later one, and idempotence needs 5 or fewer.
delivery.timeout.ms The upper bound on reporting success or failure after send() returns, covering the wait in a batch, the wait for the acknowledgement and every retry (default 120000, 2 minutes); it should be at least request.timeout.ms plus linger.ms.
enable.auto.commit Whether the consumer commits its offsets in the background every auto.commit.interval.ms (default true); with it on, records processed between the last commit and a crash are read again.
auto.commit.interval.ms How often a consumer with enable.auto.commit=true commits its offsets (default 5000, 5 seconds); records processed after the last commit are read again after a crash.
isolation.level read_uncommitted (the default) returns every record, including those of open and aborted transactions; read_committed returns only committed transactional records and stops at the last stable offset.
transactional.id A stable name that turns on transactions for a producer (and implies idempotence); a new producer that calls initTransactions() with the same id fences the older one.

Reproduce it in the simulator

Kafka enable.idempotence=false: a retry writes a duplicate (The duplicate problem) Open at the moment it happens: Kafka enable.idempotence=false: a retry writes a duplicate
At-least-once delivery in Kafka: where duplicates come from (Where duplicates come from) Open at the moment it happens: At-least-once delivery in Kafka: where duplicates come from
Kafka idempotent producer dedup vs a plain producer (Idempotent dedup vs duplicate) Open at the moment it happens: Kafka idempotent producer dedup vs a plain producer

Sources

All troubleshooting pages