Monedula

Scenario 05.1.1 · Time-based retention (retention.ms)

Kafka retention.ms and a consumer that falls behind

Storage · Intermediate · about 3 min · Apache Kafka 4.3

A consumer left behind by retention.ms: the sweep deletes the prefix it had not read, its next poll hits OFFSET_OUT_OF_RANGE, and auto.offset.reset decides where it lands.

Run this scenario in the playground

OFFSET_OUT_OF_RANGE · reset to earliest. t+2.300s

On its next poll the group fetched from offset 0, but the log now starts at 2. auto.offset.reset=earliest seeks it to logStartOffset (2), it reads the surviving 2-3 and advances to HW=4. A lagging consumer recovered by jumping past the deleted prefix.

Open at this step: step 4, OFFSET_OUT_OF_RANGE · reset to earliest

What you start with

The retention sweep itself - records older than retention.ms deleted, logStartOffset advancing - is what 00.2.3 showed. This scenario is about the consumer that was not keeping up: its group is offline while the old batch ages out, so when it comes back its position sits below the new logStartOffset. Watch what its next poll gets, and where auto.offset.reset=earliest puts it.

Step by step

  1. retention.ms armed · group offline t+0.000s A delete topic with retention.ms=1500. The consumer in group c1 has crashed (its member is offline) so it stays at offset 0. A retention_sweep loop runs every 500 ms (the broker's log.retention.check.interval.ms, 5 minutes by default). Press play to produce. Open at this step: step 1, retention.ms armed · group offline
  2. Two batches produced - different ages t+1.400s Four records produced (HW=4). Offsets 0-1 were appended at t=200; offsets 2-3 at t=1300. Each record carries its append timestamp. Nothing has aged out yet (all within the 1500 ms window), so logStartOffset is still 0. Open at this step: step 2, Two batches produced - different ages
  3. Sweep deleted the aged prefix t+2.050s At t=2000 the sweep found offsets 0-1 (age 1800 ms) older than retention.ms=1500 and advanced logStartOffset to 2 (retention_run) - they are gone, while 2-3 survive. Real Kafka deletes whole segments once their newest record is older than retention.ms, so this split assumes 0-1 sit in their own rolled segment. The consumer just restarted, but its position 0 is now below the log start. Open at this step: step 3, Sweep deleted the aged prefix
  4. OFFSET_OUT_OF_RANGE · reset to earliest t+2.300s On its next poll the group fetched from offset 0, but the log now starts at 2. auto.offset.reset=earliest seeks it to logStartOffset (2), it reads the surviving 2-3 and advances to HW=4. A lagging consumer recovered by jumping past the deleted prefix. Open at this step: step 4, OFFSET_OUT_OF_RANGE · reset to earliest
  5. The rest ages out too t+3.200s By t=3000 offsets 2-3 also crossed the retention.ms window, so the next sweep advanced logStartOffset to 4 - the log is now empty. The consumer already committed offset 4, so it stays in range. The sweep loop keeps running, deleting nothing until new records arrive. Open at this step: step 5, The rest ages out too

The takeaway

A consumer that lags past retention.ms does not get an error it can retry: the records are gone. OFFSET_OUT_OF_RANGE is the signal, and auto.offset.reset=earliest is what turns it into a silent skip to the new log start - the reset=none alternative is 05.1.3.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.1.1" title="Kafka Simulator: Time-based retention (retention.ms)" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/time-retention/">Kafka retention.ms and a consumer that falls behind</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

More in Storage

All scenarios