---
title: "Why does a Kafka consumer reprocess messages after restart?"
description: "A restarted Kafka consumer resumes from its committed offset, not its last processed record. Why that replays records, when it replays all of them, and the fix."
url: https://monedula.dev/kafka-simulator/troubleshooting/consumer-reprocesses-messages-after-restart/
---

# The consumer reprocesses messages after a restart

Kafka symptom.

## Quick answer

A restarted consumer resumes from its group's committed offset, not from the last record it processed, so everything processed after the last successful commit is delivered again. A few records is normal at-least-once behaviour. A full replay means the committed offset was missing and auto.offset.reset=earliest sent the group back to the start of the log.

## What it means

The consumer tracks two positions per partition: its position, the next offset it will read, and the committed offset, stored by the group coordinator, which is where it recovers to after a restart or when another member takes the partition over. With enable.auto.commit=true, the default, the client commits its position in the background every auto.commit.interval.ms (5 seconds by default) and once more in close(). With manual commits, the committed offset moves only when your code calls commitSync() or commitAsync(). Either way, the gap between the two positions at the moment the process stops is what gets processed again. On startup the consumer logs where it resumes: "Setting offset for partition orders-0 to the committed offset FetchPosition{offset=1500, ...}" when the group has a committed offset, or "Found no committed offset for partition orders-0" and then "Resetting offset for partition orders-0 to position FetchPosition{offset=0, ...}." when auto.offset.reset decides. The replay scenario reproduces the full-replay case: the group comes back with no committed offset and auto.offset.reset=earliest re-reads the log.

## Likely causes, most common first

1. **The process was killed instead of closed.** close() commits the current offsets when auto commit is on, and a dynamic member also leaves the group. A process stopped with SIGKILL, killed for running out of memory, or given less time to stop than close() needs (up to 30 seconds by default) skips that commit, so up to one auto-commit interval of work is replayed.
2. **Commits are coarse or failing.** Code that commits once per batch replays the whole batch after a crash. Failed commitAsync() calls are easy to miss, and a commit made after the member was evicted for exceeding max.poll.interval.ms fails with CommitFailedException, so the new owner processes that batch again.
3. **The committed offset is gone.** Offsets of a group with no members expire after offsets.retention.minutes (7 days by default). A group stopped for longer than that, a changed group.id, or a committed offset that retention has since deleted all leave the consumer without a valid offset, and auto.offset.reset decides where it starts: earliest replays the whole retained log, latest (the default) skips to the end.
4. **A rebalance handed over uncommitted work.** While an instance restarts, its partitions move to other members. If it did not commit in onPartitionsRevoked, the new owner starts from the older committed offset. When a session times out, the partitions can be reassigned before revocation runs at all.
5. **The duplicates are in the log.** If the same payload sits at two different offsets, the consumer is not reprocessing anything: a producer wrote the record twice, typically after a retry with idempotence off or a resend from application code.

## How to fix it

- Look at the resume point before and after a restart: kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group shows CURRENT-OFFSET, the committed offset the group resumes from, and prints a dash where nothing is committed.
- Shut down cleanly: from the shutdown hook call wakeup(), then close() from the polling thread, as the KafkaConsumer Javadoc shows, and give the process enough time to finish.
- Commit in onPartitionsRevoked, and commit right after processing rather than on a timer when the replay window matters.
- Make processing idempotent, since redelivery after a failure is part of the at-least-once contract. For Kafka-to-Kafka pipelines, commit the consumed offsets inside a transaction with sendOffsetsToTransaction.
- Keep group.id stable, choose auto.offset.reset on purpose, and raise offsets.retention.minutes if groups can stay stopped for longer than a week.

## What not to do

- Committing before processing, or switching to auto.offset.reset=latest, to stop the replay. Both turn duplicates into records that are never processed.

## Configs involved

- [`enable.auto.commit`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_enable.auto.commit): Whether the consumer commits its offsets in the background every auto.commit.interval.ms (default true); with it on, records processed between the last commit and a crash are read again.
- [`auto.commit.interval.ms`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_auto.commit.interval.ms): How often a consumer with enable.auto.commit=true commits its offsets (default 5000, 5 seconds); records processed after the last commit are read again after a crash.
- [`auto.offset.reset`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_auto.offset.reset): Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead.
- [`max.poll.interval.ms`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_max.poll.interval.ms): The longest a group member may go between poll() calls before it counts as failed and the group rebalances its partitions to other members (default 300000, 5 minutes); a static member keeps them until its session times out.
- [`offsets.retention.minutes`](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_offsets.retention.minutes): How long the group coordinator keeps the committed offsets of a group with no members, or of a partition the group no longer subscribes to (default 10080, 7 days); after that the group starts again from auto.offset.reset.
- [`group.id`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_group.id): The consumer group the consumer belongs to; members of a group share the partitions of the topics they subscribe to and commit their offsets under this id.

## Reproduce it in the simulator

- [Replay a Kafka topic: consumed records stay in the log](https://monedula.dev/kafka-simulator/scenarios/replay-rewind-offset/) (Replay: rewind to an earlier offset): [open at the moment it happens](https://monedula.dev/kafka-simulator/playground/?scenario=02.1.4&t=3)
- [At-least-once delivery in Kafka: where duplicates come from](https://monedula.dev/kafka-simulator/scenarios/where-duplicates-come-from/) (Where duplicates come from): [open at the moment it happens](https://monedula.dev/kafka-simulator/playground/?scenario=04.1.2&t=3)

## Sources

- [Apache Kafka 4.3 Javadoc: KafkaConsumer, offsets, consumer position and close()](https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html)
- [Apache Kafka 4.3 Javadoc: ConsumerRebalanceListener](https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html)
- [Apache Kafka 4.3: consumer config enable.auto.commit](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_enable.auto.commit)
- [Apache Kafka 4.3: consumer config auto.commit.interval.ms](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_auto.commit.interval.ms)
- [Apache Kafka 4.3: consumer config auto.offset.reset](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_auto.offset.reset)
- [Apache Kafka 4.3: broker config offsets.retention.minutes](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_offsets.retention.minutes)
- [Apache Kafka 4.3: design, message delivery semantics](https://kafka.apache.org/43/design/design/#message-delivery-semantics)
- [Apache Kafka 4.3: checking consumer position](https://kafka.apache.org/43/operations/basic-kafka-operations/#checking-consumer-position)
- [apache/kafka 4.3.0: kafka-consumer-groups.sh prints "-" for a missing offset (ConsumerGroupCommand.java)](https://github.com/apache/kafka/blob/4.3.0/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java)
- [Apache Kafka 4.3.0 source: ConsumerUtils.java (Setting offset to the committed offset)](https://github.com/apache/kafka/blob/4.3.0/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerUtils.java)
- [Apache Kafka 4.3.0 source: ConsumerCoordinator.java (Found no committed offset)](https://github.com/apache/kafka/blob/4.3.0/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java)
- [Apache Kafka 4.3.0 source: SubscriptionState.java (Resetting offset for partition)](https://github.com/apache/kafka/blob/4.3.0/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java)

## Related

- [OffsetOutOfRangeException in Kafka: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/offset-out-of-range/)
- [Kafka duplicate messages: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/duplicate-messages/)