Kafka topic config
delete.retention.ms: tombstones on compacted topics
delete.retention.ms is how long a compacted topic keeps a tombstone, a record with a null value that deletes its key, after compaction has removed the older values of that key (default 86400000, 1 day). Once it passes, the next compaction removes the tombstone too. A consumer that reads the topic from the start must finish within that window, or it may never see the delete.
- Config
delete.(topic)retention. ms - Default
- 86400000 (1 day)
- Valid values
- 0 or more (milliseconds)
- Also set at
- The broker-wide default is log.cleaner.delete.retention.ms (also 1 day).
What it does
On a topic with cleanup.policy=compact, deleting a key means producing a tombstone for it. The log cleaner keeps only the latest record per key, so the tombstone survives while the values before it are removed. delete.retention.ms is the grace period the tombstone then stays in the log, so consumers that read the topic, such as a service rebuilding a cache or a table from it, learn that the key was deleted. After the grace period the cleaner drops the tombstone and the key leaves no trace.
The trade-off
A longer delete.retention.ms gives slow or rebuilding consumers more time to see every delete, at the cost of tombstones taking space and showing up in every full read. A shorter one keeps the log lean, but a consumer that starts reading from the beginning and takes longer than the window can miss a tombstone and keep a deleted key in its state forever.
Advice
- Keep delete.retention.ms longer than the longest time a consumer needs to read the whole topic from offset 0, including restores and rebuilds.
- Remember that the clock starts at compaction, not at produce time, and compaction only touches closed segments, so tombstones usually stay longer than the setting.
- Do not set it to 0 on a topic that consumers rebuild state from.
See it in the simulator
- Kafka tombstones: deleting a key with delete.retention.ms (Tombstones & delete.retention.ms) Step 4: Tombstone lingering (grace window) Open at this step: Kafka tombstones: deleting a key with delete.retention.ms
- Kafka log compaction: cleanup.policy=compact explained (Compaction) Step 2: Compacted · older keys collapsed Open at this step: Kafka log compaction: cleanup.policy=compact explained