Scenario 03.2.2 · Leader balancing

Kafka preferred leader election after a rack failure

Replication · Advanced · about 3 min · Apache Kafka 4.3

A rack dies, leadership concentrates on survivors; restart the rack and a preferred-leader election re-spreads it.

What you start with

Each partition has a "preferred" leader - the first replica in its assignment. When a rack fails, leadership for its partitions moves onto the surviving brokers and piles up there; after the rack is restarted, a preferred-leader election re-spreads leadership back to its original distribution.

Step by step

1. Six-broker rack-aware cluster - every partition led by its preferred replica (2 leaders per rack) t+0.000s Open at this step: step 1, Six-broker rack-aware cluster - every partition led by its preferred replica (2 leaders per rack)
2. Record O1 produced and replicated across all three racks t+5.000s Open at this step: step 2, Record O1 produced and replicated across all three racks
3. rack-c dead - partitions 2 & 5 fail over to survivors; leadership now skewed onto rack-a / rack-b t+8.000s Each partition keeps ≥2 in-sync replicas (one per surviving rack), so acks=all still commits - but the two rack-c-preferred leaders are now non-preferred. Open at this step: step 3, rack-c dead - partitions 2 & 5 fail over to survivors; leadership now skewed onto rack-a / rack-b
4. rack-c restarted - broker-5/broker-6 caught up and rejoined ISR (as followers); leadership still skewed t+13.500s The preferred replicas are back in-sync, but this run has auto.leader.rebalance.enable off, so leadership stays put until a preferred-leader election runs; with Kafka's default the controller would do it within 300 s. Open at this step: step 4, rack-c restarted - broker-5/broker-6 caught up and rejoined ISR (as followers); leadership still skewed
5. Preferred-leader election ran - leadership re-spread to the preferred replicas; imbalance back to 0 t+16.500s Open at this step: step 5, Preferred-leader election ran - leadership re-spread to the preferred replicas; imbalance back to 0

The takeaway

Failures concentrate leadership on survivors; a preferred-leader election rebalances it once brokers return, keeping any single broker from carrying a disproportionate share of leader traffic.

Configs in this scenario

• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.2.2" title="Kafka Simulator: Leader balancing" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/leader-balancing/">Kafka preferred leader election after a rack failure</a></p>

Further reading

Apache Kafka documentation

More in Replication

• Kafka ISR (in-sync replicas): how replicas leave and rejoin
• min.insync.replicas with acks=all: when Kafka refuses writes
• Kafka slow follower: replica.lag.time.max.ms and the ISR
• Kafka high watermark pinned by a slow in-sync replica
• Losing two Kafka brokers with min.insync.replicas=2
• Kafka unclean.leader.election.enable: offline or data loss
• Kafka unclean leader election: permanent loss of acked data
• Kafka log truncation when an old leader returns
• Kafka broker.rack: rack-aware replica placement
• Kafka rack-aware reassignment after adding a rack
• Increase the replication factor of an existing Kafka topic

All scenarios