Kafka topic config

cleanup.policy: delete vs compact

Topic config · default delete · Apache Kafka 4.3

cleanup.policy decides how a Kafka topic gets rid of old records. delete (the default) removes whole segments once they pass retention.ms or retention.bytes. compact keeps at least the latest record for every key and removes older ones, so the topic works as a changelog. compact,delete runs both: the latest value per key, but only within the retention window.

Config: cleanup.policy (topic)
Default: delete
Valid values: delete, compact, or both as a list (compact,delete)
Also set at: The broker-wide default for topics that do not set it is log.cleanup.policy.
Reference: Apache Kafka documentation for cleanup.policy

Scenario 05.2.1 · Compaction step 2

Compacted · older keys collapsed. Each key keeps only its latest offset; offsets 0-2 are removed and survivors keep their original offsets (gaps). The user-2 tombstone lingers during delete.retention.ms. Real Kafka never compacts the active segment, so these records would first have to roll into a closed one. Open this step in the full playground

Values

delete Time and size retention: segments older than retention.ms, or beyond retention.bytes, are deleted whatever their keys.
compact Log compaction: the log cleaner rewrites closed segments to keep only the latest record per key; a record with a null value (a tombstone) deletes its key after delete.retention.ms.
compact,delete Both: compaction removes superseded values, and retention still deletes segments past retention.ms or retention.bytes, even if they hold the only value of a key.

What it does

With delete, the log is a stream with a time or size window: consumers that read within it see every record. With compact, the log is a keyed table: a consumer that reads from the start rebuilds the latest state of every key, which is how Kafka itself stores consumer offsets in __consumer_offsets and how Kafka Streams backs up state stores. Compaction never touches the active segment, and min.compaction.lag.ms and min.cleanable.dirty.ratio (0.5 by default) decide how soon closed segments are cleaned, so older values of a key stay readable for a while.

The trade-off

delete bounds disk by time or size but forgets keys that have not been written recently. compact keeps every live key forever, at the cost of log cleaner CPU and I/O and of disk that grows with the number of keys, and it needs every record to have a key. compact,delete bounds both, but a key not updated within retention.ms disappears.

Advice

Use compact for changelogs, reference data and anything a consumer rebuilds state from, and produce a tombstone to delete a key.
Use delete for event streams where old events lose their value after a while.
A compacted topic cannot take records without a key: the broker rejects them.

See it in the simulator

Kafka log compaction: cleanup.policy=compact explained (Compaction) Step 2: Compacted · older keys collapsed Open at this step: Kafka log compaction: cleanup.policy=compact explained
Kafka cleanup.policy=compact,delete: both policies at once (compact,delete policy) Step 4: Compacted · stale A=a1 collapsed Open at this step: Kafka cleanup.policy=compact,delete: both policies at once

Related configs

All Kafka configs