Kafka topic config

retention.bytes: size-based retention

Topic config · default -1 (no size limit) · Apache Kafka 4.3

retention.bytes is the size a partition of a topic with cleanup.policy=delete may grow to before Kafka deletes its oldest log segments (default -1, no size limit). The limit is per partition, not per topic. When retention.ms is also set, whichever limit a segment hits first deletes it.

Config: retention.bytes (topic)
Default: -1 (no size limit)
Valid values: -1, or a size in bytes
Also set at: The broker-wide default for topics that do not set it is log.retention.bytes (also -1).
Reference: Apache Kafka documentation for retention.bytes

Scenario 05.1.2 · Size-based retention (retention.bytes) step 3

Over the cap - oldest records evicted. A second batch pushed the log to 6 records (6144 bytes, HW=6) - over the 3072-byte cap. The sweep at t=1000 deleted the OLDEST records (offsets 0-2) until the log fit again, advancing logStartOffset to 3. Their age was irrelevant: they were written a second ago. Open this step in the full playground

What it does

A partition log is a series of segment files. Every log.retention.check.interval.ms (5 minutes by default) the broker checks each partition, and while the partition is larger than retention.bytes it deletes the oldest segment, moving the log start offset forward. The age of the records does not matter: on a busy topic, records written a minute ago go if they are the oldest. The active segment is never deleted, so a partition can stay above the limit by up to one segment.bytes (1 GiB by default).

The trade-off

A size limit protects broker disks from a traffic spike that time-based retention alone would let through, but it makes how long data is kept depend on throughput: twice the traffic halves the time a slow consumer has to read a record before it is gone. Leaving it at -1 keeps retention predictable in time and leaves disk sizing to you.

Advice

Size disks from retention.bytes times the partition count times the replication factor, plus one segment per partition of headroom.
Use retention.bytes as a safety cap next to retention.ms, not as the main retention rule, unless a fixed disk budget per topic matters more than a fixed time window.
Lower segment.bytes on a small topic with a small size cap, or the active segment alone can exceed it.

See it in the simulator

Kafka retention.bytes: size-based log retention (Size-based retention (retention.bytes)) Step 3: Over the cap - oldest records evicted Open at this step: Kafka retention.bytes: size-based log retention

Related configs

All Kafka configs