Scenario 05.2.3 · compact,delete policy

Kafka cleanup.policy=compact,delete: both policies at once

Storage · Intermediate · about 3 min · Apache Kafka 4.3

A topic running both policies: compaction collapses duplicate keys while time retention independently ages out the old log prefix.

Run this scenario in the playground

Retention · aged-out B=b1 deleted. t+2.100s

Retention sweep - the DELETE half. With retention.ms=1500, B=b1 (timestamped ~100ms) is now too old, so the aged prefix is deleted and logStartOffset advances past it, even though b1 was B's only record. Real Kafka deletes whole segments by their newest timestamp, so this assumes b1's segment holds nothing newer. A=a2 and C=c1 survive both policies.

Open at this step: step 5, Retention · aged-out B=b1 deleted

What you start with

cleanup.policy=compact,delete runs both cleanups on one topic at once: compaction collapses duplicate keys down to their latest value, while time-based retention independently ages out and deletes the old log prefix.

Step by step

compact,delete topic (empty) t+0.000s A topic with cleanup.policy=compact,delete runs BOTH policies. Compaction collapses to the latest value per key; retention.ms (1500ms) independently deletes records older than the window. The two are orthogonal. Open at this step: step 1, compact,delete topic (empty)
Old batch · A=a1, B=b1 t+0.200s The old batch lands: A=a1 at offset 0 and B=b1 at offset 1, both timestamped ~100ms. They are still young relative to the 1500ms retention window. Open at this step: step 2, Old batch · A=a1, B=b1
Recent batch · A→a2, C=c1 t+1.620s A recent batch updates A to a2 (offset 2) and adds C=c1 (offset 3). Key A now has two records in the log - a compaction target - while B and C have one each. Open at this step: step 3, Recent batch · A→a2, C=c1
Compacted · stale A=a1 collapsed t+1.700s Compaction pass - the COMPACT half. A collapses to its latest value a2; the stale A=a1 at offset 0 is removed (note the offset gap). B and C are untouched - compaction never deletes a key's latest record. Real Kafka never compacts the active segment, so these records would first have to roll into a closed one. Open at this step: step 4, Compacted · stale A=a1 collapsed
Retention · aged-out B=b1 deleted t+2.100s Retention sweep - the DELETE half. With retention.ms=1500, B=b1 (timestamped ~100ms) is now too old, so the aged prefix is deleted and logStartOffset advances past it, even though b1 was B's only record. Real Kafka deletes whole segments by their newest timestamp, so this assumes b1's segment holds nothing newer. A=a2 and C=c1 survive both policies. Open at this step: step 5, Retention · aged-out B=b1 deleted

The takeaway

The two policies are independent - compaction bounds the log by distinct keys while delete-retention still trims by age, so old records leave even when their key holds the latest value.

Configs in this scenario

cleanup.policy Whether old data is removed by age/size (delete), by keeping only the latest value per key (compact), or both - it decides whether the topic behaves like a stream or a keyed changelog. How cleanup.policy works
retention.ms How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.2.3" title="Kafka Simulator: compact,delete policy" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/compact-delete-policy/">Kafka cleanup.policy=compact,delete: both policies at once</a></p>

More in Storage

All scenarios