Scenario 03.2.3 · Unclean leader
Kafka unclean.leader.election.enable: offline or data loss
A stale survivor is promoted with data loss when unclean.leader.election.enable is flipped on.
Run this scenario in the playgroundCluster ready - RF 3, min.ISR 2. t+0.000s
orders has 3 in-sync replicas. unclean.leader.election.enable is OFF: Kafka prefers durability over availability.
Open at this step: step 1, Cluster ready - RF 3, min.ISR 22 records committed on all three. t+4.100s
High-watermark is 2 - broker-3 holds both records.
Open at this step: step 2, 2 records committed on all threebroker-3 throttled out of ISR - stale survivor. t+12.100s
broker-3 is stuck at LEO 2 while the leader committed up to 5; it has fallen out of ISR but is still alive.
Open at this step: step 3, broker-3 throttled out of ISR - stale survivorbroker-1 killed - broker-2 leads (clean election). t+16.600s
broker-2 is the only in-sync replica left, so it is elected cleanly with no data loss.
Open at this step: step 4, broker-1 killed - broker-2 leads (clean election)broker-2 killed - partition OFFLINE. t+21.000s
ISR is empty. With unclean election disabled, Kafka refuses to promote the stale broker-3: durable, but unavailable.
Open at this step: step 5, broker-2 killed - partition OFFLINEUnclean election enabled - broker-3 leads, records lost. t+25.000s
broker-3 (LEO 2) is promoted unclean. The high-watermark drops from 5 to 2 - 3 committed records are gone. Availability restored at the cost of durability. (The simulator elects at once; in KRaft a dynamically enabled flag waits for the periodic check, 5 min by default, unless you run kafka-leader-election.sh with the unclean option.)
Open at this step: step 6, Unclean election enabled - broker-3 leads, records lostWhat you start with
With unclean.leader.election.enable=false, a partition whose entire ISR is dead stays offline rather than promote a stale replica. Here both in-sync brokers are killed after a follower has been throttled out of date, then the unclean flag is flipped on.
Step by step
- Cluster ready - RF 3, min.ISR 2 t+0.000s orders has 3 in-sync replicas. unclean.leader.election.enable is OFF: Kafka prefers durability over availability. Open at this step: step 1, Cluster ready - RF 3, min.ISR 2
- 2 records committed on all three t+4.100s High-watermark is 2 - broker-3 holds both records. Open at this step: step 2, 2 records committed on all three
- broker-3 throttled out of ISR - stale survivor t+12.100s broker-3 is stuck at LEO 2 while the leader committed up to 5; it has fallen out of ISR but is still alive. Open at this step: step 3, broker-3 throttled out of ISR - stale survivor
- broker-1 killed - broker-2 leads (clean election) t+16.600s broker-2 is the only in-sync replica left, so it is elected cleanly with no data loss. Open at this step: step 4, broker-1 killed - broker-2 leads (clean election)
- broker-2 killed - partition OFFLINE t+21.000s ISR is empty. With unclean election disabled, Kafka refuses to promote the stale broker-3: durable, but unavailable. Open at this step: step 5, broker-2 killed - partition OFFLINE
- Unclean election enabled - broker-3 leads, records lost t+25.000s broker-3 (LEO 2) is promoted unclean. The high-watermark drops from 5 to 2 - 3 committed records are gone. Availability restored at the cost of durability. (The simulator elects at once; in KRaft a dynamically enabled flag waits for the periodic check, 5 min by default, unless you run kafka-leader-election.sh with the unclean option.) Open at this step: step 6, Unclean election enabled - broker-3 leads, records lost
The takeaway
Flipping unclean.leader.election.enable on promotes the stale survivor and brings the partition back online, but at the cost of losing every record it never received - availability bought with data loss. (In real KRaft the election is not instant: a dynamically enabled flag waits for the controller's periodic check, 5 min by default, unless kafka-leader-election.sh runs an unclean election.)
Configs in this scenario
-
unclean.Lets a replica that is not in the ISR become leader when no in-sync replica is left, which brings the partition back online but loses the writes that replica never received (default false). How unclean.leader.election.enable worksleader. election. enable
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.2.3" title="Kafka Simulator: Unclean leader" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/unclean-leader/">Kafka unclean.leader.election.enable: offline or data loss</a></p>Further reading
Troubleshooting
More in Replication
- Kafka ISR (in-sync replicas): how replicas leave and rejoin
- min.insync.replicas with acks=all: when Kafka refuses writes
- Kafka slow follower: replica.lag.time.max.ms and the ISR
- Kafka high watermark pinned by a slow in-sync replica
- Losing two Kafka brokers with min.insync.replicas=2
- Kafka preferred leader election after a rack failure
- Kafka unclean leader election: permanent loss of acked data
- Kafka log truncation when an old leader returns
- Kafka broker.rack: rack-aware replica placement
- Kafka rack-aware reassignment after adding a rack
- Increase the replication factor of an existing Kafka topic