Scenario 05.1.2 · Size-based retention (retention.bytes)

Kafka retention.bytes: size-based log retention

Storage · Intermediate · about 2 min · Apache Kafka 4.3

When the log grows past retention.bytes, a scheduled sweep deletes the OLDEST records until it fits again, advancing logStartOffset - no matter how recent they are.

Run this scenario in the playground

Over again - two more gone. t+1.600s

A third batch (e6, e7) took the log to 5 records - over the cap again - and the sweep at t=1500 evicted two more (offsets 3-4), moving logStartOffset to 5. This is the rhythm of size-based retention: every time the head outgrows retention.bytes the tail loses whatever is oldest, and the log hovers at the cap. Real Kafka deletes whole segments rather than single records, so the boundary moves in coarser steps.

Open at this step: step 4, Over again - two more gone

What you start with

retention.bytes caps the total size of a partition log rather than its age. When the log grows past the limit, the retention sweep deletes the oldest records until it fits again, advancing logStartOffset. Three batches push the log over the cap twice, so you can watch the same sweep evict records that are seconds old.

Step by step

retention.bytes armed t+0.000s A delete topic with retention.bytes=3072. The engine sizes the log at a fixed 1024 bytes/record, so the cap admits 3 records. A retention_sweep loop runs every 500 ms (the broker's log.retention.check.interval.ms, 5 minutes by default) and deletes the OLDEST records whenever the log is over the cap. Press play to produce. Open at this step: step 1, retention.bytes armed
First batch - log exactly at the cap t+0.600s Three records produced (HW=3) = 3072 bytes, exactly retention.bytes. The first sweep (t=500) found the log fits under the cap, so it deleted nothing - logStartOffset is still 0. Open at this step: step 2, First batch - log exactly at the cap
Over the cap - oldest records evicted t+1.100s A second batch pushed the log to 6 records (6144 bytes, HW=6) - over the 3072-byte cap. The sweep at t=1000 deleted the OLDEST records (offsets 0-2) until the log fit again, advancing logStartOffset to 3. Their age was irrelevant: they were written a second ago. Open at this step: step 3, Over the cap - oldest records evicted
Over again - two more gone t+1.600s A third batch (e6, e7) took the log to 5 records - over the cap again - and the sweep at t=1500 evicted two more (offsets 3-4), moving logStartOffset to 5. This is the rhythm of size-based retention: every time the head outgrows retention.bytes the tail loses whatever is oldest, and the log hovers at the cap. Real Kafka deletes whole segments rather than single records, so the boundary moves in coarser steps. Open at this step: step 4, Over again - two more gone

The takeaway

Size-based retention bounds disk usage, not record age - once a partition exceeds retention.bytes its oldest records are dropped no matter how recent they are, and it keeps happening every time the log outgrows the cap.

Configs in this scenario

retention.bytes The size a partition may grow to before its oldest segments are deleted under cleanup.policy=delete (default -1, no size limit); the limit is per partition, not per topic. How retention.bytes works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.1.2" title="Kafka Simulator: Size-based retention (retention.bytes)" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/size-retention/">Kafka retention.bytes: size-based log retention</a></p>

More in Storage

All scenarios