Scenario 10.3.2 · Infinite retention on the remote tier
Kafka tiered storage with infinite retention (Redpanda)
local.retention.ms auto-offloads aged local segments to the remote tier (localStartOffset rises) while infinite retention.ms keeps everything remote - a lagging consumer reads the old offsets back from remote, nothing lost.
Run this scenario in the playgroundTiered topic · consumer offline. t+0.000s
The topic 'events' has remote.storage.enable=true, local.retention.ms=700 and a blank retention.ms, which the simulator treats as infinite total retention (in Kafka that is retention.ms=-1; an unset retention.ms falls back to the broker default of 7 days). The remote tier keeps everything and logStartOffset will never advance. Consumer c1 reads from earliest and is taken offline (kill_consumer) so it builds up lag while local segments are offloaded. Nothing produced yet.
Open at this step: step 1, Tiered topic · consumer offlineOld batch produced - all in the local log. t+0.700s
Producer p1 writes an OLD batch of 6 records (offsets 0-5). The high watermark is 6 and every record is still on local disk: localStartOffset is unset, so the whole log is local. Because retention.ms is infinite, logStartOffset stays 0.
Open at this step: step 2, Old batch produced - all in the local logOld prefix auto-offloaded to remote. t+1.100s
The scheduled local_retention_sweep finds the batch older than local.retention.ms (700 ms) and offloads [0,6) to the remote tier - an AUTOMATIC tier_offload that raises localStartOffset to 6. The records are not deleted: logStartOffset is still 0, so all six survive in the remote tier. This is the key contrast with total retention.ms, which would instead advance logStartOffset and delete them.
Open at this step: step 3, Old prefix auto-offloaded to remoteRecent batch lands - local again. t+1.300s
A RECENT batch (offsets 6-8) is produced, so the high watermark is 9. These newest records sit on local disk (localStartOffset 6 ≤ offset < 9) while the older 0-5 are remote-only - exactly the split local.retention.ms creates: recent data local, older data remote, none of it lost.
Open at this step: step 4, Recent batch lands - local againLagging consumer reads offset 0 from remote. t+2.600s
The consumer is back (restart_consumer) and polls from offset 0. Because 0 is below localStartOffset, those records are served from the REMOTE tier and the engine emits a remote_read; max.poll.records=3 advances the position to 3. Meanwhile a later sweep found the recent batch older than local.retention.ms too, so Kafka rolled the active segment, copied it to remote and removed it locally: localStartOffset is now 9 and the whole log lives in remote, still fully readable.
Open at this step: step 5, Lagging consumer reads offset 0 from remoteStill below localStartOffset - remote again. t+3.400s
The next poll starts at offset 3 - still below localStartOffset (9) - so it is served from remote again (a second remote_read). The position climbs to 6 as the consumer steadily drains the backlog out of the remote tier.
Open at this step: step 6, Still below localStartOffset - remote againCaught up to the high watermark - nothing lost. t+4.200s
The final poll reads offsets 6-8 from remote (a third remote_read) and the consumer reaches the high watermark (position 9). Everything the producer wrote was delivered even though none of it remained on local disk (Kafka rolls a tiered topic's active segment once it breaches local retention, then uploads it): infinite retention.ms kept it all in the remote tier, and local.retention.ms bounded only local storage - it never costs data.
Open at this step: step 7, Caught up to the high watermark - nothing lostWhat you start with
With tiered storage, local.retention.ms governs only how long segments stay on broker disk: aged segments are offloaded to the remote tier (localStartOffset rises) while an infinite retention.ms keeps every record in the cluster forever. A consumer lagging below localStartOffset transparently reads those old offsets back from the remote tier. The simulator uses Apache Kafka's KIP-405 names; on Redpanda the same windows are retention.local.target.ms (local) and retention.ms (total), with tiering switched on by redpanda.storage.mode=tiered (Redpanda 26.1+; earlier redpanda.remote.write and redpanda.remote.read) once cloud_storage_enabled is set on the cluster.
Step by step
- Tiered topic · consumer offline t+0.000s The topic 'events' has remote.storage.enable=true, local.retention.ms=700 and a blank retention.ms, which the simulator treats as infinite total retention (in Kafka that is retention.ms=-1; an unset retention.ms falls back to the broker default of 7 days). The remote tier keeps everything and logStartOffset will never advance. Consumer c1 reads from earliest and is taken offline (kill_consumer) so it builds up lag while local segments are offloaded. Nothing produced yet. Open at this step: step 1, Tiered topic · consumer offline
- Old batch produced - all in the local log t+0.700s Producer p1 writes an OLD batch of 6 records (offsets 0-5). The high watermark is 6 and every record is still on local disk: localStartOffset is unset, so the whole log is local. Because retention.ms is infinite, logStartOffset stays 0. Open at this step: step 2, Old batch produced - all in the local log
- Old prefix auto-offloaded to remote t+1.100s The scheduled local_retention_sweep finds the batch older than local.retention.ms (700 ms) and offloads [0,6) to the remote tier - an AUTOMATIC tier_offload that raises localStartOffset to 6. The records are not deleted: logStartOffset is still 0, so all six survive in the remote tier. This is the key contrast with total retention.ms, which would instead advance logStartOffset and delete them. Open at this step: step 3, Old prefix auto-offloaded to remote
- Recent batch lands - local again t+1.300s A RECENT batch (offsets 6-8) is produced, so the high watermark is 9. These newest records sit on local disk (localStartOffset 6 ≤ offset < 9) while the older 0-5 are remote-only - exactly the split local.retention.ms creates: recent data local, older data remote, none of it lost. Open at this step: step 4, Recent batch lands - local again
- Lagging consumer reads offset 0 from remote t+2.600s The consumer is back (restart_consumer) and polls from offset 0. Because 0 is below localStartOffset, those records are served from the REMOTE tier and the engine emits a remote_read; max.poll.records=3 advances the position to 3. Meanwhile a later sweep found the recent batch older than local.retention.ms too, so Kafka rolled the active segment, copied it to remote and removed it locally: localStartOffset is now 9 and the whole log lives in remote, still fully readable. Open at this step: step 5, Lagging consumer reads offset 0 from remote
- Still below localStartOffset - remote again t+3.400s The next poll starts at offset 3 - still below localStartOffset (9) - so it is served from remote again (a second remote_read). The position climbs to 6 as the consumer steadily drains the backlog out of the remote tier. Open at this step: step 6, Still below localStartOffset - remote again
- Caught up to the high watermark - nothing lost t+4.200s The final poll reads offsets 6-8 from remote (a third remote_read) and the consumer reaches the high watermark (position 9). Everything the producer wrote was delivered even though none of it remained on local disk (Kafka rolls a tiered topic's active segment once it breaches local retention, then uploads it): infinite retention.ms kept it all in the remote tier, and local.retention.ms bounded only local storage - it never costs data. Open at this step: step 7, Caught up to the high watermark - nothing lost
The takeaway
Separating local from total retention lets you keep broker disks small and cheap while retaining data indefinitely - old offsets are never lost, just served more slowly from the remote tier.
Configs in this scenario
-
local.On a tiered-storage topic, how long a segment stays on the broker disk once it has been copied to remote storage (default -2, which means retention.ms); the data stays readable from the remote tier until retention.ms removes it.retention. ms -
retention.How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms worksms -
remote.Turns on tiered storage (KIP-405) for the topic: older log segments are offloaded from local disk to a remote tier once they pass local.retention.ms, and a consumer reading past them hits a remote read. Total data is still bounded by retention.ms - tiering changes WHERE the data lives, not how long it is kept.storage. enable -
max.The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.poll. records
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.3.2" title="Kafka Simulator: Infinite retention on the remote tier" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/infinite-retention-tiered/">Kafka tiered storage with infinite retention (Redpanda)</a></p>