Kafka troubleshooting
The consumer reprocesses messages after a restart
A restarted consumer resumes from its group's committed offset, not from the last record it processed, so everything processed after the last successful commit is delivered again. A few records is normal at-least-once behaviour. A full replay means the committed offset was missing and auto.offset.reset=earliest sent the group back to the start of the log.
Reproduce it in the playground3 steps beforeg1 has read O1-O3 · committed offset 3. t+2.200s
g1 polled all three records and auto-commit saved its position, so the committed offset is 3 - the bookmark says "resume after O3". The records are still in the log; reading them deleted nothing.
Open at this step: step 1, g1 has read O1-O3 · committed offset 32 steps beforeGroup stopped to reset its offset. t+2.500s
To rewind a real group you first stop it: kafka-consumer-groups --reset-offsets refuses to run while members are active. Stopping alone keeps the committed offset of 3; here the group is deleted, which also deletes its offsets, so we can choose a new starting point.
Open at this step: step 2, Group stopped to reset its offset1 step beforeg1 rewound to the start · about to replay. t+2.800s
The same group id is back with auto.offset.reset=earliest and no committed offset, so it seeks to the log start (offset 0). Its position dropped from 3 back to 0 - it is about to re-read records it already processed. This is replay: reprocess after a bug fix, rebuild a derived view, etc.
Open at this step: step 3, g1 rewound to the start · about to replayThe moment it happensRe-reading O1-O3 again. t+3.200s
One record per poll (max.poll.records=1), g1 works back through the same offsets it consumed before - position is 2, so O1 and O2 have been delivered a second time. The log was never consumed-away; the offset is just a movable bookmark.
Open at this step: step 4, Re-reading O1-O3 againRight afterReplay complete · back at offset 3. t+3.700s
Having re-read O1-O3, g1's position and committed offset are 3 again - exactly where they were before the rewind, but every record was delivered twice. (Replay is at-least-once: a downstream that can't tolerate duplicates must dedupe.)
Open at this step: step 5, Replay complete · back at offset 3What it means
The consumer tracks two positions per partition: its position, the next offset it will read, and the committed offset, stored by the group coordinator, which is where it recovers to after a restart or when another member takes the partition over. With enable.auto.commit=true, the default, the client commits its position in the background every auto.commit.interval.ms (5 seconds by default) and once more in close(). With manual commits, the committed offset moves only when your code calls commitSync() or commitAsync(). Either way, the gap between the two positions at the moment the process stops is what gets processed again. On startup the consumer logs where it resumes: "Setting offset for partition orders-0 to the committed offset FetchPosition{offset=1500, ...}" when the group has a committed offset, or "Found no committed offset for partition orders-0" and then "Resetting offset for partition orders-0 to position FetchPosition{offset=0, ...}." when auto.offset.reset decides. The replay scenario reproduces the full-replay case: the group comes back with no committed offset and auto.offset.reset=earliest re-reads the log.
Likely causes, most common first
- The process was killed instead of closed. close() commits the current offsets when auto commit is on, and a dynamic member also leaves the group. A process stopped with SIGKILL, killed for running out of memory, or given less time to stop than close() needs (up to 30 seconds by default) skips that commit, so up to one auto-commit interval of work is replayed.
- Commits are coarse or failing. Code that commits once per batch replays the whole batch after a crash. Failed commitAsync() calls are easy to miss, and a commit made after the member was evicted for exceeding max.poll.interval.ms fails with CommitFailedException, so the new owner processes that batch again.
- The committed offset is gone. Offsets of a group with no members expire after offsets.retention.minutes (7 days by default). A group stopped for longer than that, a changed group.id, or a committed offset that retention has since deleted all leave the consumer without a valid offset, and auto.offset.reset decides where it starts: earliest replays the whole retained log, latest (the default) skips to the end.
- A rebalance handed over uncommitted work. While an instance restarts, its partitions move to other members. If it did not commit in onPartitionsRevoked, the new owner starts from the older committed offset. When a session times out, the partitions can be reassigned before revocation runs at all.
- The duplicates are in the log. If the same payload sits at two different offsets, the consumer is not reprocessing anything: a producer wrote the record twice, typically after a retry with idempotence off or a resend from application code.
How to fix it
- Look at the resume point before and after a restart: kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group shows CURRENT-OFFSET, the committed offset the group resumes from, and prints a dash where nothing is committed.
- Shut down cleanly: from the shutdown hook call wakeup(), then close() from the polling thread, as the KafkaConsumer Javadoc shows, and give the process enough time to finish.
- Commit in onPartitionsRevoked, and commit right after processing rather than on a timer when the replay window matters.
- Make processing idempotent, since redelivery after a failure is part of the at-least-once contract. For Kafka-to-Kafka pipelines, commit the consumed offsets inside a transaction with sendOffsetsToTransaction.
- Keep group.id stable, choose auto.offset.reset on purpose, and raise offsets.retention.minutes if groups can stay stopped for longer than a week.
What not to do
- Committing before processing, or switching to auto.offset.reset=latest, to stop the replay. Both turn duplicates into records that are never processed.
Configs involved
enable.Whether the consumer commits its offsets in the background every auto.commit.interval.ms (default true); with it on, records processed between the last commit and a crash are read again.auto. commit auto.How often a consumer with enable.auto.commit=true commits its offsets (default 5000, 5 seconds); records processed after the last commit are read again after a crash.commit. interval. ms auto.Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead.offset. reset max.The longest a group member may go between poll() calls before it counts as failed and the group rebalances its partitions to other members (default 300000, 5 minutes); a static member keeps them until its session times out.poll. interval. ms offsets.How long the group coordinator keeps the committed offsets of a group with no members, or of a partition the group no longer subscribes to (default 10080, 7 days); after that the group starts again from auto.offset.reset.retention. minutes group.The consumer group the consumer belongs to; members of a group share the partitions of the topics they subscribe to and commit their offsets under this id.id
Reproduce it in the simulator
- Replay a Kafka topic: consumed records stay in the log (Replay: rewind to an earlier offset) Open at the moment it happens: Replay a Kafka topic: consumed records stay in the log
- At-least-once delivery in Kafka: where duplicates come from (Where duplicates come from) Open at the moment it happens: At-least-once delivery in Kafka: where duplicates come from
Sources
- Apache Kafka 4.3 Javadoc: KafkaConsumer, offsets, consumer position and close()
- Apache Kafka 4.3 Javadoc: ConsumerRebalanceListener
- Apache Kafka 4.3: consumer config enable.auto.commit
- Apache Kafka 4.3: consumer config auto.commit.interval.ms
- Apache Kafka 4.3: consumer config auto.offset.reset
- Apache Kafka 4.3: broker config offsets.retention.minutes
- Apache Kafka 4.3: design, message delivery semantics
- Apache Kafka 4.3: checking consumer position
- apache/kafka 4.3.0: kafka-consumer-groups.sh prints "-" for a missing offset (ConsumerGroupCommand.java)
- Apache Kafka 4.3.0 source: ConsumerUtils.java (Setting offset to the committed offset)
- Apache Kafka 4.3.0 source: ConsumerCoordinator.java (Found no committed offset)
- Apache Kafka 4.3.0 source: SubscriptionState.java (Resetting offset for partition)