Scenario 05.2.1 · Compaction
Kafka log compaction: cleanup.policy=compact explained
cleanup.policy=compact · keep the latest value per key, purge tombstones after the grace window.
Run this scenario in the playgroundDirty log · 6 records, 3 keys. t+0.000s
A compacted topic. Six records; user-1 appears three times, user-2 ends with a tombstone. Press play to compact.
Open at this step: step 1, Dirty log · 6 records, 3 keysCompacted · older keys collapsed. t+0.600s
Each key keeps only its latest offset; offsets 0-2 are removed and survivors keep their original offsets (gaps). The user-2 tombstone lingers during delete.retention.ms. Real Kafka never compacts the active segment, so these records would first have to roll into a closed one.
Open at this step: step 2, Compacted · older keys collapsedTombstone purged. t+1.600s
After the grace window the tombstone is removed too. Only the latest record per live key remains.
Open at this step: step 3, Tombstone purgedWhat you start with
cleanup.policy=compact retains records by key instead of deleting by age. A compaction run rewrites the log, keeping only the latest value for each key, dropping the superseded versions, and purging tombstones once their grace window passes.
Step by step
- Dirty log · 6 records, 3 keys t+0.000s A compacted topic. Six records; user-1 appears three times, user-2 ends with a tombstone. Press play to compact. Open at this step: step 1, Dirty log · 6 records, 3 keys
- Compacted · older keys collapsed t+0.600s Each key keeps only its latest offset; offsets 0-2 are removed and survivors keep their original offsets (gaps). The user-2 tombstone lingers during delete.retention.ms. Real Kafka never compacts the active segment, so these records would first have to roll into a closed one. Open at this step: step 2, Compacted · older keys collapsed
- Tombstone purged t+1.600s After the grace window the tombstone is removed too. Only the latest record per live key remains. Open at this step: step 3, Tombstone purged
The takeaway
Compaction turns a topic into a key→latest-value log - duplicate keys collapse to their most recent value, so log size tracks the number of distinct keys, not total write volume.
Configs in this scenario
-
cleanup.Whether old data is removed by age/size (delete), by keeping only the latest value per key (compact), or both - it decides whether the topic behaves like a stream or a keyed changelog. How cleanup.policy workspolicy -
delete.On a compacted topic, how long a tombstone (a record with a null value) stays in the log after compaction before it is removed too (default 86400000, 1 day); a consumer reading from the start must finish within it to see every delete. How delete.retention.ms worksretention. ms
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.2.1" title="Kafka Simulator: Compaction" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/compaction/">Kafka log compaction: cleanup.policy=compact explained</a></p>Further reading
More in Storage
- Kafka retention.ms and a consumer that falls behind
- Kafka retention.bytes: size-based log retention
- Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none
- Kafka tombstones: deleting a key with delete.retention.ms
- Kafka cleanup.policy=compact,delete: both policies at once
- Kafka tiered storage: offloading old segments to remote
- Kafka tiered storage: a consumer reads from the remote tier
- Kafka local.retention.ms vs retention.ms in tiered storage
- Diskless Kafka topics (KIP-1163): the write path
- Diskless vs classic Kafka: cross-AZ replication traffic
- Diskless Kafka produce latency: the price of object storage