--- title: "Kafka local.retention.ms vs retention.ms in tiered storage" description: "local.retention.ms offloads old segments to remote, retention.ms deletes the oldest: one Kafka partition ends with deleted, remote-only and local ranges." url: https://monedula.dev/kafka-simulator/scenarios/local-vs-total-retention/ --- # Kafka local.retention.ms vs retention.ms in tiered storage Scenario 05.3.3 · Local vs total retention. Storage · Advanced · about 4 min · Apache Kafka 4.3. local.retention.ms offloads aged segments to the remote tier (localStartOffset rises) while a longer, FINITE retention.ms later deletes the oldest prefix from remote too (logStartOffset advances) - leaving three regions: deleted, remote-only, and local. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3 ## What you start with A tiered topic carries two retention windows on one log. The short local.retention.ms offloads aged segments to the remote tier (raising localStartOffset) without losing anything, while a longer but FINITE total retention.ms eventually deletes the oldest prefix from the remote tier as well (advancing logStartOffset). The result is three regions on the same partition: deleted, remote-only, and local. ## Step by step 1. **Tiered topic · two retention windows** (t+0.000s): The topic 'events' has remote.storage.enable=true with TWO windows: local.retention.ms=900 (local disk) and a longer, FINITE retention.ms=2200 (total/remote). The consumer c1 reads from earliest and is taken offline (kill_consumer) so it lags while the windows age the log. Nothing produced yet. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=0) 2. **Old batch produced - all local** (t+0.300s): Producer p1 writes the OLD batch (offsets 0-2). The high watermark is 3 and every record is on local disk: localStartOffset is unset and logStartOffset is 0, so the whole log is local. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=1) 3. **Middle batch lands - still all local** (t+1.100s): A MIDDLE batch (offsets 3-5) brings the high watermark to 6. Both batches are still on local disk - neither retention window has fired on this data yet. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=2) 4. **Oldest batch offloaded to remote (localStartOffset rises)** (t+1.700s): The scheduled local_retention_sweep finds the old batch older than local.retention.ms (900 ms) and offloads [0,3) to the remote tier (the simulator works per batch; Kafka moves whole segments, so this assumes the old batch sits in its own rolled segment), raising localStartOffset to 3 ABOVE logStartOffset (still 0). The records are NOT deleted: they are remote-only and still readable. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=3) 5. **Whole log now remote - nothing deleted yet** (t+2.300s): A later local sweep finds the middle batch older than local.retention.ms too, so Kafka rolls the active segment, copies it to remote and removes it locally: localStartOffset reaches 6 (= the high watermark) and the entire log lives in the remote tier, with only an empty active segment on disk. Total retention.ms has NOT fired yet, so logStartOffset is still 0 and every record survives. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=4) 6. **Three regions: total retention deletes the oldest prefix** (t+2.600s): The recent batch (offsets 6-8) lands (high watermark 9) AND the scheduled retention_sweep, hitting the longer retention.ms=2200 window, deletes [0,3) from the remote tier - advancing logStartOffset to 3. Now logStartOffset(3) < localStartOffset(6) < highWatermark(9): three regions - deleted [0,3), remote-only [3,6), local [6,9). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=5) 7. **Lagging consumer: remote-only read back, deleted prefix lost** (t+3.400s): The consumer is back and polls from offset 0. Offset 0 is below logStartOffset(3), so it gets offset_out_of_range and resets to earliest (logStartOffset 3); it then reads the remote-only prefix [3,6) back from remote (remote_read) and the local [6,9) to reach the high watermark - recovering everything that survived but having permanently lost the deleted offsets 0-2. The three regions still hold: logStartOffset 3 < localStartOffset 6 < highWatermark 9. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=05.3.3&t=6) ## The takeaway local.retention.ms and retention.ms are independent: the first only relocates data to cheaper remote storage (still readable via remote_read), the second is the real durability bound - once retention.ms passes, the oldest records are deleted everywhere and a consumer reading below logStartOffset gets offset_out_of_range. ## Configs in this scenario - [`local.retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_local.retention.ms): On a tiered-storage topic, how long a segment stays on the broker disk once it has been copied to remote storage (default -2, which means retention.ms); the data stays readable from the remote tier until retention.ms removes it. - [`retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms): How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. [How retention.ms works](https://monedula.dev/kafka-simulator/configs/retention-ms/) - [`remote.storage.enable`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_remote.storage.enable): Turns on tiered storage (KIP-405) for the topic: older log segments are offloaded from local disk to a remote tier once they pass local.retention.ms, and a consumer reading past them hits a remote read. Total data is still bounded by retention.ms - tiering changes WHERE the data lives, not how long it is kept. ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/operations/tiered-storage/) ## Troubleshooting - [OffsetOutOfRangeException in Kafka: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/offset-out-of-range/)