--- title: "Kafka unclean leader election: permanent loss of acked data" description: "With unclean.leader.election.enable=true in the topic config, a stale replica is promoted on its own and three acked Kafka records are lost for good." url: https://monedula.dev/kafka-simulator/scenarios/unclean-election-data-loss/ --- # Kafka unclean leader election: permanent loss of acked data Scenario 03.2.4 · Unclean election: permanent data loss. Replication · Advanced · about 4 min · Apache Kafka 4.3. unclean.leader.election.enable=true turns a double broker failure into a permanent consistency hole - committed records vanish and never return. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4 ## What you start with This is the consistency hole behind unclean.leader.election.enable=true: records acknowledged to an acks=all producer (the HW advanced past them, so they are committed) can still be lost. Here the topic config turns the flag ON (Kafka's default is false); a stale follower is frozen, three more records commit on the survivors, then both in-sync brokers die. ## Step by step 1. **Cluster ready - unclean.leader.election is ON** (t+0.000s): orders has RF 3 / min.ISR 2, but its topic config sets unclean.leader.election.enable=true (the default is false) - a latent hazard. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=0) 2. **2 records committed on all three** (t+4.100s): High-watermark is 2 - every replica holds both records. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=1) 3. **broker-3 throttled out of ISR - stale at LEO 2** (t+12.100s): broker-3 froze at LEO 2 and fell out of ISR. Three more records committed on broker-1 + broker-2, so HW is 5: with acks=all the producer was told records 2-4 are durable - but they live on only two replicas. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=2) 4. **broker-1 killed - broker-2 leads cleanly (no loss yet)** (t+16.600s): The leader died, but broker-2 is still in sync at LEO 5, so it is elected cleanly. The committed records survive - on a single replica now. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=3) 5. **broker-2 killed - stale broker-3 promoted UNCLEAN, committed records lost** (t+20.600s): ISR is empty and the only live replica is the stale broker-3 (LEO 2). Unclean election promotes it anyway: the high-watermark drops from 5 to 2. The three records the producer was told were committed (2-4) are gone - availability bought with a permanent consistency hole. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=4) 6. **Loss is permanent - HW stays 2 under the stale leader** (t+24.000s): Records 2-4 survive only on the two DEAD brokers; the live cluster's high-watermark is pinned at 2 under broker-3. Even when the old leaders come back they must follow the new leader's shorter log - the committed records never return. This is the cost of unclean.leader.election.enable=true. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.4&t=5) ## The takeaway When the only live replica is a stale one, unclean election auto-promotes it and the high watermark truncates - committed records survive only on the dead brokers and never come back. The loss is permanent. ## Configs in this scenario - [`unclean.leader.election.enable`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_unclean.leader.election.enable): Lets a replica that is not in the ISR become leader when no in-sync replica is left, which brings the partition back online but loses the writes that replica never received (default false). [How unclean.leader.election.enable works](https://monedula.dev/kafka-simulator/configs/unclean-leader-election-enable/) - [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/) - [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/) ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_unclean.leader.election.enable) ## Troubleshooting - [Kafka messages lost after broker failure: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/messages-lost-after-broker-failure/)