Kafka consumer config
auto.offset.reset: earliest vs latest
auto.offset.reset decides where a consumer starts reading when its group has no committed offset for a partition, or the committed offset no longer exists in the log. latest (the default) starts at the end and reads only records produced from then on; earliest starts at the oldest record still retained and reads the whole backlog. It does nothing for a group that already has a valid committed offset.
- Config
auto.(consumer)offset. reset - Default
- latest
- Valid values
- latest, earliest, by_duration:<ISO 8601 duration> (such as by_duration:PT1H), none
Values
latestStart at the log end offset. Records already in the partition are skipped; the consumer sees only what is produced after it was assigned the partition.earliestStart at the log start offset, the oldest record retention has not deleted yet, and read everything from there.by_Start at the first record with a timestamp at most that long ago, such as by_duration:P1D for the last day. Added in Kafka 4.0 (KIP-1106).duration:<duration> nonePick nothing: poll() throws NoOffsetForPartitionException when there is no committed offset, or OffsetOutOfRangeException when it is out of range, and the application decides.
What it does
A consumer group resumes each partition from its committed offset. auto.offset.reset is only the fallback for the two cases where that offset is missing or useless: a group that has never committed for the partition (a new group, a newly added partition, or a group whose offsets expired after offsets.retention.minutes, 7 days by default, with no members), and a committed offset that retention has already deleted, which the broker answers with OFFSET_OUT_OF_RANGE. Restarting a consumer, deploying a new version or rebalancing does not apply it as long as the group still has a valid committed offset, so changing the setting on an existing group usually changes nothing.
The trade-off
earliest never skips data, but a new group reads the whole retained log first, which can be days of records to process before it catches up, and a group recovering from an out-of-range offset reprocesses whatever is left. latest starts instantly and never replays, but it silently skips everything produced before the group first joined, and when partitions are added to a topic, records producers write to a new partition before the group is assigned it are skipped too. none makes the choice explicit at the cost of handling the exception yourself.
Advice
- Use earliest for a group that must see every record, such as one that builds a table or a search index from the topic; latest suits groups that only care about new events, such as live alerting.
- To move an existing group, reset its offsets with kafka-consumer-groups.sh --reset-offsets while the group is stopped; changing auto.offset.reset does not move a group that has committed offsets.
- Treat an OFFSET_OUT_OF_RANGE reset as a symptom: the group fell behind retention.ms and lost records it never read. Alert on consumer lag before it happens.
See it in the simulator
- Kafka auto.offset.reset: earliest vs latest vs none (auto.offset.reset) Step 2: g-latest starts at the END · g-earliest at the START Open at this step: Kafka auto.offset.reset: earliest vs latest vs none
- Kafka retention.ms and a consumer that falls behind (Time-based retention (retention.ms)) Step 4: OFFSET_OUT_OF_RANGE · reset to earliest Open at this step: Kafka retention.ms and a consumer that falls behind
- Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none (Offset out of range) Step 3: OFFSET_OUT_OF_RANGE · group stuck Open at this step: Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none