Monedula

Scenario 05.3.3 · Local vs total retention

Kafka local.retention.ms vs retention.ms in tiered storage

Storage · Advanced · about 4 min · Apache Kafka 4.3

local.retention.ms offloads aged segments to the remote tier (localStartOffset rises) while a longer, FINITE retention.ms later deletes the oldest prefix from remote too (logStartOffset advances) - leaving three regions: deleted, remote-only, and local.

Run this scenario in the playground

Lagging consumer: remote-only read back, deleted prefix lost. t+3.400s

The consumer is back and polls from offset 0. Offset 0 is below logStartOffset(3), so it gets offset_out_of_range and resets to earliest (logStartOffset 3); it then reads the remote-only prefix [3,6) back from remote (remote_read) and the local [6,9) to reach the high watermark - recovering everything that survived but having permanently lost the deleted offsets 0-2. The three regions still hold: logStartOffset 3 < localStartOffset 6 < highWatermark 9.

Open at this step: step 7, Lagging consumer: remote-only read back, deleted prefix lost

What you start with

A tiered topic carries two retention windows on one log. The short local.retention.ms offloads aged segments to the remote tier (raising localStartOffset) without losing anything, while a longer but FINITE total retention.ms eventually deletes the oldest prefix from the remote tier as well (advancing logStartOffset). The result is three regions on the same partition: deleted, remote-only, and local.

Step by step

  1. Tiered topic · two retention windows t+0.000s The topic 'events' has remote.storage.enable=true with TWO windows: local.retention.ms=900 (local disk) and a longer, FINITE retention.ms=2200 (total/remote). The consumer c1 reads from earliest and is taken offline (kill_consumer) so it lags while the windows age the log. Nothing produced yet. Open at this step: step 1, Tiered topic · two retention windows
  2. Old batch produced - all local t+0.300s Producer p1 writes the OLD batch (offsets 0-2). The high watermark is 3 and every record is on local disk: localStartOffset is unset and logStartOffset is 0, so the whole log is local. Open at this step: step 2, Old batch produced - all local
  3. Middle batch lands - still all local t+1.100s A MIDDLE batch (offsets 3-5) brings the high watermark to 6. Both batches are still on local disk - neither retention window has fired on this data yet. Open at this step: step 3, Middle batch lands - still all local
  4. Oldest batch offloaded to remote (localStartOffset rises) t+1.700s The scheduled local_retention_sweep finds the old batch older than local.retention.ms (900 ms) and offloads [0,3) to the remote tier (the simulator works per batch; Kafka moves whole segments, so this assumes the old batch sits in its own rolled segment), raising localStartOffset to 3 ABOVE logStartOffset (still 0). The records are NOT deleted: they are remote-only and still readable. Open at this step: step 4, Oldest batch offloaded to remote (localStartOffset rises)
  5. Whole log now remote - nothing deleted yet t+2.300s A later local sweep finds the middle batch older than local.retention.ms too, so Kafka rolls the active segment, copies it to remote and removes it locally: localStartOffset reaches 6 (= the high watermark) and the entire log lives in the remote tier, with only an empty active segment on disk. Total retention.ms has NOT fired yet, so logStartOffset is still 0 and every record survives. Open at this step: step 5, Whole log now remote - nothing deleted yet
  6. Three regions: total retention deletes the oldest prefix t+2.600s The recent batch (offsets 6-8) lands (high watermark 9) AND the scheduled retention_sweep, hitting the longer retention.ms=2200 window, deletes [0,3) from the remote tier - advancing logStartOffset to 3. Now logStartOffset(3) < localStartOffset(6) < highWatermark(9): three regions - deleted [0,3), remote-only [3,6), local [6,9). Open at this step: step 6, Three regions: total retention deletes the oldest prefix
  7. Lagging consumer: remote-only read back, deleted prefix lost t+3.400s The consumer is back and polls from offset 0. Offset 0 is below logStartOffset(3), so it gets offset_out_of_range and resets to earliest (logStartOffset 3); it then reads the remote-only prefix [3,6) back from remote (remote_read) and the local [6,9) to reach the high watermark - recovering everything that survived but having permanently lost the deleted offsets 0-2. The three regions still hold: logStartOffset 3 < localStartOffset 6 < highWatermark 9. Open at this step: step 7, Lagging consumer: remote-only read back, deleted prefix lost

The takeaway

local.retention.ms and retention.ms are independent: the first only relocates data to cheaper remote storage (still readable via remote_read), the second is the real durability bound - once retention.ms passes, the oldest records are deleted everywhere and a consumer reading below logStartOffset gets offset_out_of_range.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.3.3" title="Kafka Simulator: Local vs total retention" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/local-vs-total-retention/">Kafka local.retention.ms vs retention.ms in tiered storage</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

More in Storage

All scenarios