Scenario 03.2.4 · Unclean election: permanent data loss

Kafka unclean leader election: permanent loss of acked data

Replication · Advanced · about 4 min · Apache Kafka 4.3

unclean.leader.election.enable=true turns a double broker failure into a permanent consistency hole - committed records vanish and never return.

What you start with

This is the consistency hole behind unclean.leader.election.enable=true: records acknowledged to an acks=all producer (the HW advanced past them, so they are committed) can still be lost. Here the topic config turns the flag ON (Kafka's default is false); a stale follower is frozen, three more records commit on the survivors, then both in-sync brokers die.

Step by step

1. Cluster ready - unclean.leader.election is ON t+0.000s orders has RF 3 / min.ISR 2, but its topic config sets unclean.leader.election.enable=true (the default is false) - a latent hazard. Open at this step: step 1, Cluster ready - unclean.leader.election is ON
2. 2 records committed on all three t+4.100s High-watermark is 2 - every replica holds both records. Open at this step: step 2, 2 records committed on all three
3. broker-3 throttled out of ISR - stale at LEO 2 t+12.100s broker-3 froze at LEO 2 and fell out of ISR. Three more records committed on broker-1 + broker-2, so HW is 5: with acks=all the producer was told records 2-4 are durable - but they live on only two replicas. Open at this step: step 3, broker-3 throttled out of ISR - stale at LEO 2
4. broker-1 killed - broker-2 leads cleanly (no loss yet) t+16.600s The leader died, but broker-2 is still in sync at LEO 5, so it is elected cleanly. The committed records survive - on a single replica now. Open at this step: step 4, broker-1 killed - broker-2 leads cleanly (no loss yet)
5. broker-2 killed - stale broker-3 promoted UNCLEAN, committed records lost t+20.600s ISR is empty and the only live replica is the stale broker-3 (LEO 2). Unclean election promotes it anyway: the high-watermark drops from 5 to 2. The three records the producer was told were committed (2-4) are gone - availability bought with a permanent consistency hole. Open at this step: step 5, broker-2 killed - stale broker-3 promoted UNCLEAN, committed records lost
6. Loss is permanent - HW stays 2 under the stale leader t+24.000s Records 2-4 survive only on the two DEAD brokers; the live cluster's high-watermark is pinned at 2 under broker-3. Even when the old leaders come back they must follow the new leader's shorter log - the committed records never return. This is the cost of unclean.leader.election.enable=true. Open at this step: step 6, Loss is permanent - HW stays 2 under the stale leader

The takeaway

When the only live replica is a stale one, unclean election auto-promotes it and the high watermark truncates - committed records survive only on the dead brokers and never come back. The loss is permanent.

Configs in this scenario

• unclean.leader.election.enable Lets a replica that is not in the ISR become leader when no in-sync replica is left, which brings the partition back online but loses the writes that replica never received (default false). How unclean.leader.election.enable works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.2.4" title="Kafka Simulator: Unclean election: permanent data loss" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/unclean-election-data-loss/">Kafka unclean leader election: permanent loss of acked data</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• Kafka messages lost after broker failure: causes and fixes

More in Replication

• Kafka ISR (in-sync replicas): how replicas leave and rejoin
• min.insync.replicas with acks=all: when Kafka refuses writes
• Kafka slow follower: replica.lag.time.max.ms and the ISR
• Kafka high watermark pinned by a slow in-sync replica
• Losing two Kafka brokers with min.insync.replicas=2
• Kafka preferred leader election after a rack failure
• Kafka unclean.leader.election.enable: offline or data loss
• Kafka log truncation when an old leader returns
• Kafka broker.rack: rack-aware replica placement
• Kafka rack-aware reassignment after adding a rack
• Increase the replication factor of an existing Kafka topic

All scenarios