Kafka topic config

retention.ms: how long Kafka keeps records

Topic config · default 604800000 (7 days) · Apache Kafka 4.3

retention.ms is how long a topic with cleanup.policy=delete keeps a log segment before Kafka may delete it (default 604800000, 7 days). A segment is deleted once its newest record is older than retention.ms, so records can live longer than the limit. -1 keeps records forever.

Config: retention.ms (topic)
Default: 604800000 (7 days)
Valid values: -1 (no time limit), or a duration in milliseconds
Also set at: The broker-wide default for topics that do not set it is log.retention.ms, falling back to log.retention.minutes and then log.retention.hours (168, 7 days).
Reference: Apache Kafka documentation for retention.ms

Scenario 05.1.1 · Time-based retention (retention.ms) step 3

Sweep deleted the aged prefix. At t=2000 the sweep found offsets 0-1 (age 1800 ms) older than retention.ms=1500 and advanced logStartOffset to 2 (retention_run) - they are gone, while 2-3 survive. Real Kafka deletes whole segments once their newest record is older than retention.ms, so this split assumes 0-1 sit in their own rolled segment. The consumer just restarted, but its position 0 is now below the log start. Open this step in the full playground

What it does

Every log.retention.check.interval.ms (5 minutes by default) the broker deletes the segments of each partition whose largest record timestamp is older than retention.ms, moving the log start offset forward. Retention works on whole segments, and the active segment is only deleted after it rolls, which happens at segment.bytes (1 GiB) or segment.ms (7 days), so a quiet topic can keep records well past retention.ms. A consumer whose committed offset falls below the new log start gets OFFSET_OUT_OF_RANGE and resets according to auto.offset.reset.

The trade-off

Longer retention lets consumers fall further behind, and lets you replay or bootstrap new consumers from history, at the cost of disk on every replica. Shorter retention saves disk but turns any consumer outage longer than the window into lost data for that consumer. Tiered storage moves the trade-off: local.retention.ms keeps recent data on broker disk while retention.ms governs the total.

Advice

Set retention.ms longer than the longest consumer outage you want to survive without losing records, with margin for a weekend.
Set it per topic rather than relying on the broker default, so a change to the default cannot silently shorten a topic retention.
For data you must keep forever, use -1 with tiered storage, or a compacted topic if only the latest value per key matters.

See it in the simulator

Kafka retention.ms and a consumer that falls behind (Time-based retention (retention.ms)) Step 3: Sweep deleted the aged prefix Open at this step: Kafka retention.ms and a consumer that falls behind
Kafka cleanup.policy=compact,delete: both policies at once (compact,delete policy) Step 5: Retention · aged-out B=b1 deleted Open at this step: Kafka cleanup.policy=compact,delete: both policies at once

Related configs

All Kafka configs