---
title: "Kafka tiered storage with infinite retention (Redpanda)"
description: "With retention.ms=-1 the remote tier keeps everything: old segments offload after 700 ms and a lagging Kafka consumer reads all nine records from remote."
url: https://monedula.dev/kafka-simulator/scenarios/infinite-retention-tiered/
---

# Kafka tiered storage with infinite retention (Redpanda)

Scenario 10.3.2 · Infinite retention on the remote tier. Tiered storage · Advanced · about 4 min · Apache Kafka 4.3 · Redpanda.

local.retention.ms auto-offloads aged local segments to the remote tier (localStartOffset rises) while infinite retention.ms keeps everything remote - a lagging consumer reads the old offsets back from remote, nothing lost.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2

## What you start with

With tiered storage, local.retention.ms governs only how long segments stay on broker disk: aged segments are offloaded to the remote tier (localStartOffset rises) while an infinite retention.ms keeps every record in the cluster forever. A consumer lagging below localStartOffset transparently reads those old offsets back from the remote tier. The simulator uses Apache Kafka's KIP-405 names; on Redpanda the same windows are retention.local.target.ms (local) and retention.ms (total), with tiering switched on by redpanda.storage.mode=tiered (Redpanda 26.1+; earlier redpanda.remote.write and redpanda.remote.read) once cloud_storage_enabled is set on the cluster.

## Step by step

1. **Tiered topic · consumer offline** (t+0.000s): The topic 'events' has remote.storage.enable=true, local.retention.ms=700 and a blank retention.ms, which the simulator treats as infinite total retention (in Kafka that is retention.ms=-1; an unset retention.ms falls back to the broker default of 7 days). The remote tier keeps everything and logStartOffset will never advance. Consumer c1 reads from earliest and is taken offline (kill_consumer) so it builds up lag while local segments are offloaded. Nothing produced yet. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=0)
2. **Old batch produced - all in the local log** (t+0.700s): Producer p1 writes an OLD batch of 6 records (offsets 0-5). The high watermark is 6 and every record is still on local disk: localStartOffset is unset, so the whole log is local. Because retention.ms is infinite, logStartOffset stays 0. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=1)
3. **Old prefix auto-offloaded to remote** (t+1.100s): The scheduled local_retention_sweep finds the batch older than local.retention.ms (700 ms) and offloads [0,6) to the remote tier - an AUTOMATIC tier_offload that raises localStartOffset to 6. The records are not deleted: logStartOffset is still 0, so all six survive in the remote tier. This is the key contrast with total retention.ms, which would instead advance logStartOffset and delete them. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=2)
4. **Recent batch lands - local again** (t+1.300s): A RECENT batch (offsets 6-8) is produced, so the high watermark is 9. These newest records sit on local disk (localStartOffset 6 ≤ offset < 9) while the older 0-5 are remote-only - exactly the split local.retention.ms creates: recent data local, older data remote, none of it lost. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=3)
5. **Lagging consumer reads offset 0 from remote** (t+2.600s): The consumer is back (restart_consumer) and polls from offset 0. Because 0 is below localStartOffset, those records are served from the REMOTE tier and the engine emits a remote_read; max.poll.records=3 advances the position to 3. Meanwhile a later sweep found the recent batch older than local.retention.ms too, so Kafka rolled the active segment, copied it to remote and removed it locally: localStartOffset is now 9 and the whole log lives in remote, still fully readable. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=4)
6. **Still below localStartOffset - remote again** (t+3.400s): The next poll starts at offset 3 - still below localStartOffset (9) - so it is served from remote again (a second remote_read). The position climbs to 6 as the consumer steadily drains the backlog out of the remote tier. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=5)
7. **Caught up to the high watermark - nothing lost** (t+4.200s): The final poll reads offsets 6-8 from remote (a third remote_read) and the consumer reaches the high watermark (position 9). Everything the producer wrote was delivered even though none of it remained on local disk (Kafka rolls a tiered topic's active segment once it breaches local retention, then uploads it): infinite retention.ms kept it all in the remote tier, and local.retention.ms bounded only local storage - it never costs data. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=10.3.2&t=6)

## The takeaway

Separating local from total retention lets you keep broker disks small and cheap while retaining data indefinitely - old offsets are never lost, just served more slowly from the remote tier.

## Configs in this scenario

- [`local.retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_local.retention.ms): On a tiered-storage topic, how long a segment stays on the broker disk once it has been copied to remote storage (default -2, which means retention.ms); the data stays readable from the remote tier until retention.ms removes it.
- [`retention.ms`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms): How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. [How retention.ms works](https://monedula.dev/kafka-simulator/configs/retention-ms/)
- [`remote.storage.enable`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_remote.storage.enable): Turns on tiered storage (KIP-405) for the topic: older log segments are offloaded from local disk to a remote tier once they pass local.retention.ms, and a consumer reading past them hits a remote read. Total data is still bounded by retention.ms - tiering changes WHERE the data lives, not how long it is kept.
- [`max.poll.records`](https://kafka.apache.org/43/configuration/consumer-configs/#consumerconfigs_max.poll.records): The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.

## Further reading

- [Redpanda documentation](https://docs.redpanda.com/current/manage/tiered-storage/)