Scenario 10.3.1 · Confluent storage lifecycle: offload, read-back, expire
Kafka tiered storage lifecycle: offload, read back, expire
remote.storage.enable + local.retention.ms + a FINITE retention.ms - walk a tiered topic through its full lifecycle: offload to remote (localStartOffset rises), read an old offset back from remote, then let total retention delete the oldest remote prefix (logStartOffset advances).
Run this scenario in the playgroundTiered topic · finite retention · consumer offline. t+0.000s
The topic 'events' has remote.storage.enable=true, local.retention.ms=700 (broker disk window) and a FINITE retention.ms=3000 (total window). These are Apache Kafka's KIP-405 names; Confluent Platform's own Tiered Storage names the local window confluent.tier.local.hotset.ms and keeps retention.ms as the total. Consumer c1 reads from earliest and is taken offline (kill_consumer) so it builds up lag while local segments are offloaded to the remote tier. Nothing produced yet.
Open at this step: step 1, Tiered topic · finite retention · consumer offlineProduce - old batch lands on local disk. t+0.700s
Producer p1 writes an OLD batch of 6 records (offsets 0-5). The high watermark is 6 and every record is still on local broker disk: localStartOffset is unset, so the whole log is local. logStartOffset is 0 - nothing offloaded or expired yet.
Open at this step: step 2, Produce - old batch lands on local diskOffload - old prefix moved to the remote tier. t+1.100s
local.retention.ms (700 ms) elapses for the old batch: the scheduled local_retention_sweep offloads [0,6) to the remote tier - an automatic tier_offload that raises localStartOffset to 6. Local disk shrinks, but the data is NOT lost: logStartOffset is still 0, so all six records survive in remote. This is the offload step of the lifecycle.
Open at this step: step 3, Offload - old prefix moved to the remote tierProduce - recent batch is local again. t+1.300s
A RECENT batch (offsets 6-8) is produced, so the high watermark is 9. These newest records sit on local disk (localStartOffset 6 ≤ offset < 9) while the older 0-5 are remote-only - the split local.retention.ms creates: recent data local, older data remote, none of it lost.
Open at this step: step 4, Produce - recent batch is local againRead-back - lagging consumer reads an old offset from remote. t+2.600s
The consumer is back (restart_consumer) and polls from offset 0. Because 0 is below localStartOffset, those records are served from the REMOTE tier and the engine emits a remote_read; max.poll.records=3 advances the position. A later local sweep found the recent batch older than local.retention.ms too, so Kafka rolled the active segment, copied it to remote and removed it locally: localStartOffset is now 9 and the whole log lives in remote, with only an empty active segment left on disk. logStartOffset is still 0: nothing expired yet.
Open at this step: step 5, Read-back - lagging consumer reads an old offset from remoteExpire - total retention deletes the oldest remote prefix. t+3.700s
The FINITE total retention.ms=3000 elapses for the OLD batch (offsets 0-5): the scheduled retention_sweep advances logStartOffset to 6 and the truly-aged remote prefix is deleted for good. The younger recent batch (6-8) is still inside the total window and survives, and the consumer - having already read the old offsets back - has caught up to the high watermark. The full tiered lifecycle: produce → offload → read-back → expire.
Open at this step: step 6, Expire - total retention deletes the oldest remote prefixWhat you start with
Tiered storage splits a topic’s lifecycle across two retention windows. local.retention.ms governs broker disk: aged segments are automatically offloaded to the remote tier (localStartOffset rises) so local disk stays small, while the data remains fully readable. A consumer lagging below localStartOffset transparently reads those offsets back from remote. The FINITE total retention.ms is the system of record: when it elapses for the oldest records, the remote prefix is deleted for good and logStartOffset advances. The simulator shows this with Apache Kafka’s KIP-405 names (remote.storage.enable, local.retention.ms). Confluent Platform’s own Tiered Storage is a different feature: it switches tiering on with confluent.tier.enable (broker confluent.tier.feature) and names the local window confluent.tier.local.hotset.ms, while retention.ms remains the total window.
Step by step
- Tiered topic · finite retention · consumer offline t+0.000s The topic 'events' has remote.storage.enable=true, local.retention.ms=700 (broker disk window) and a FINITE retention.ms=3000 (total window). These are Apache Kafka's KIP-405 names; Confluent Platform's own Tiered Storage names the local window confluent.tier.local.hotset.ms and keeps retention.ms as the total. Consumer c1 reads from earliest and is taken offline (kill_consumer) so it builds up lag while local segments are offloaded to the remote tier. Nothing produced yet. Open at this step: step 1, Tiered topic · finite retention · consumer offline
- Produce - old batch lands on local disk t+0.700s Producer p1 writes an OLD batch of 6 records (offsets 0-5). The high watermark is 6 and every record is still on local broker disk: localStartOffset is unset, so the whole log is local. logStartOffset is 0 - nothing offloaded or expired yet. Open at this step: step 2, Produce - old batch lands on local disk
- Offload - old prefix moved to the remote tier t+1.100s local.retention.ms (700 ms) elapses for the old batch: the scheduled local_retention_sweep offloads [0,6) to the remote tier - an automatic tier_offload that raises localStartOffset to 6. Local disk shrinks, but the data is NOT lost: logStartOffset is still 0, so all six records survive in remote. This is the offload step of the lifecycle. Open at this step: step 3, Offload - old prefix moved to the remote tier
- Produce - recent batch is local again t+1.300s A RECENT batch (offsets 6-8) is produced, so the high watermark is 9. These newest records sit on local disk (localStartOffset 6 ≤ offset < 9) while the older 0-5 are remote-only - the split local.retention.ms creates: recent data local, older data remote, none of it lost. Open at this step: step 4, Produce - recent batch is local again
- Read-back - lagging consumer reads an old offset from remote t+2.600s The consumer is back (restart_consumer) and polls from offset 0. Because 0 is below localStartOffset, those records are served from the REMOTE tier and the engine emits a remote_read; max.poll.records=3 advances the position. A later local sweep found the recent batch older than local.retention.ms too, so Kafka rolled the active segment, copied it to remote and removed it locally: localStartOffset is now 9 and the whole log lives in remote, with only an empty active segment left on disk. logStartOffset is still 0: nothing expired yet. Open at this step: step 5, Read-back - lagging consumer reads an old offset from remote
- Expire - total retention deletes the oldest remote prefix t+3.700s The FINITE total retention.ms=3000 elapses for the OLD batch (offsets 0-5): the scheduled retention_sweep advances logStartOffset to 6 and the truly-aged remote prefix is deleted for good. The younger recent batch (6-8) is still inside the total window and survives, and the consumer - having already read the old offsets back - has caught up to the high watermark. The full tiered lifecycle: produce → offload → read-back → expire. Open at this step: step 6, Expire - total retention deletes the oldest remote prefix
The takeaway
The two windows separate concerns: local.retention.ms keeps broker disk small by offloading to remote, while a finite total retention.ms bounds how long data lives anywhere. Old offsets stay readable from remote right up until the total window expires them - the complete offload → read-back → expire lifecycle.
Configs in this scenario
-
remote.Turns on tiered storage (KIP-405) for the topic: older log segments are offloaded from local disk to a remote tier once they pass local.retention.ms, and a consumer reading past them hits a remote read. Total data is still bounded by retention.ms - tiering changes WHERE the data lives, not how long it is kept.storage. enable -
local.On a tiered-storage topic, how long a segment stays on the broker disk once it has been copied to remote storage (default -2, which means retention.ms); the data stays readable from the remote tier until retention.ms removes it.retention. ms -
retention.How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms worksms -
max.The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.poll. records
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=10.3.1" title="Kafka Simulator: Confluent storage lifecycle: offload, read-back, expire" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/confluent-tiered-lifecycle/">Kafka tiered storage lifecycle: offload, read back, expire</a></p>