---
title: "Kafka retention.ms and a consumer that falls behind"
description: "Retention deletes offsets 0-1 while a group is down. Its next poll hits OFFSET_OUT_OF_RANGE and auto.offset.reset=earliest jumps it to the new Kafka log start."
url: https://monedula.dev/kafka-simulator/scenarios/time-retention/
---

# Kafka retention.ms and a consumer that falls behind

Scenario 05.1.1 · Time-based retention (retention.ms). Storage · Intermediate · about 3 min · Apache Kafka 4.3.

A consumer left behind by retention.ms: the sweep deletes the prefix it had not read, its next poll hits OFFSET_OUT_OF_RANGE, and auto.offset.reset decides where it lands.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1

## What you start with

The retention sweep itself - records older than retention.ms deleted, logStartOffset advancing - is what 00.2.3 showed. This scenario is about the consumer that was not keeping up: its group is offline while the old batch ages out, so when it comes back its position sits below the new logStartOffset. Watch what its next poll gets, and where auto.offset.reset=earliest puts it.

## Step by step

1. **retention.ms armed · group offline** (t+0.000s): A delete topic with retention.ms=1500. The consumer in group c1 has crashed (its member is offline) so it stays at offset 0. A retention_sweep loop runs every 500 ms (the broker's log.retention.check.interval.ms, 5 minutes by default). Press play to produce. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1&t=0)
2. **Two batches produced - different ages** (t+1.400s): Four records produced (HW=4). Offsets 0-1 were appended at t=200; offsets 2-3 at t=1300. Each record carries its append timestamp. Nothing has aged out yet (all within the 1500 ms window), so logStartOffset is still 0. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1&t=1)
3. **Sweep deleted the aged prefix** (t+2.050s): At t=2000 the sweep found offsets 0-1 (age 1800 ms) older than retention.ms=1500 and advanced logStartOffset to 2 (retention_run) - they are gone, while 2-3 survive. Real Kafka deletes whole segments once their newest record is older than retention.ms, so this split assumes 0-1 sit in their own rolled segment. The consumer just restarted, but its position 0 is now below the log start. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1&t=2)
4. **OFFSET_OUT_OF_RANGE · reset to earliest** (t+2.300s): On its next poll the group fetched from offset 0, but the log now starts at 2. auto.offset.reset=earliest seeks it to logStartOffset (2), it reads the surviving 2-3 and advances to HW=4. A lagging consumer recovered by jumping past the deleted prefix. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1&t=3)
5. **The rest ages out too** (t+3.200s): By t=3000 offsets 2-3 also crossed the retention.ms window, so the next sweep advanced logStartOffset to 4 - the log is now empty. The consumer already committed offset 4, so it stays in range. The sweep loop keeps running, deleting nothing until new records arrive. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.1.1&t=4)

## The takeaway

A consumer that lags past retention.ms does not get an error it can retry: the records are gone. OFFSET_OUT_OF_RANGE is the signal, and auto.offset.reset=earliest is what turns it into a silent skip to the new log start - the reset=none alternative is 05.1.3.

## Configs in this scenario

- [`retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms): How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. [How retention.ms works](https://monedula.dev/kafka-simulator/configs/retention-ms/)
- [`auto.offset.reset`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_auto.offset.reset): Where a consumer starts when its group has no committed offset or the offset no longer exists: latest (the default), earliest, by_duration (a time back from now, such as by_duration:PT1H), or none to throw an exception instead. [How auto.offset.reset works](https://monedula.dev/kafka-simulator/configs/auto-offset-reset/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms)

## Troubleshooting

- [OffsetOutOfRangeException in Kafka: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/offset-out-of-range/)