Kafka troubleshooting

Partitions are offline with no leader

Kafka symptom

A partition is offline when it has no leader, so it serves no reads and no writes. That happens when every replica allowed to lead it is down (with unclean leader election off, only in-sync or eligible leader replicas qualify) or when the KRaft controller quorum has lost its majority and cannot elect anyone. Bring back a broker from the partition's last ISR or restore the controller quorum; an unclean election is the last resort and loses data.

Reproduce it in the playground

Scenario 03.2.3 · Unclean leader Seen in 2 more scenarios

The moment it happensbroker-2 killed - partition OFFLINE. t+21.000s

ISR is empty. With unclean election disabled, Kafka refuses to promote the stale broker-3: durable, but unavailable.

Open at this step: step 5, broker-2 killed - partition OFFLINE

What it means

Partition leaders are chosen by the active controller. When a leader fails, the controller elects another replica from the ISR, so committed records survive the failover. If no ISR member is alive, Kafka by default waits for one to return rather than promote a replica that may be missing committed records. Until then kafka-topics.sh --describe shows Leader: none for the partition, the controller metric OfflinePartitionsCount is above zero, metadata requests return LEADER_NOT_AVAILABLE for it, and producers hold its records until delivery.timeout.ms expires and then fail them with "Expiring 1 record(s) for orders-0:120001 ms has passed since batch creation". Kafka also tracks Eligible Leader Replicas (KIP-966; per the Kafka 4.3 docs, on by default for clusters created with 4.1 or later): replicas outside the ISR that are safe to lead because the high watermark cannot advance while the ISR is below min.insync.replicas. When the ISR is empty, the controller may elect one of them, and after that the last known leader if it comes back.

Likely causes, most common first

Every in-sync replica is down. Two or three broker failures can take out every member of a partition's ISR, especially when followers had already dropped out of it before the leader died. With unclean.leader.election.enable=false, the default, the out-of-sync survivors are not allowed to lead, so the partition stays offline.
Every replica is down. A topic with replication factor 1 goes offline whenever its only broker does, and topics are created that way when default.replication.factor is left at its default of 1. Replicas placed in one rack or zone go down together.
The controller quorum has lost its majority. In KRaft, elections need an active controller, and the controller quorum stays available only while a majority of controllers is alive: 3 controllers tolerate one failure, 5 tolerate two. Without a majority, a partition whose leader dies gets no new leader, even if healthy replicas exist.
Brokers are running but fenced. A broker counts as alive only while it heartbeats to the controller within broker.session.timeout.ms (9 seconds by default). A broker that cannot reach the controllers, after a network split or with a wrong controller listener, is not eligible to lead.

How to fix it

List the affected partitions with kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions, which shows every partition whose leader is missing or not live, with its Replicas and Isr.
Bring back a broker from the partition's last ISR. Once it is running and registered with the controller, it is elected leader and the partition returns with its committed data.
If many partitions went offline at once while brokers look healthy, check the quorum with kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status, and restore controllers until a majority is back.
Only when no in-sync replica can be recovered, and losing its recent records is acceptable, run an unclean election for the affected partitions: kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type unclean --topic orders --partition 0. It only acts on partitions that have no leader.
Prevent the next one: replication factor 3, broker.rack across failure domains, and 3 or 5 controllers spread across those domains too.

What not to do

Turning on unclean.leader.election.enable for the whole cluster to end the outage. It then applies to every later failure as well, and when it is set dynamically in KRaft it waits for the periodic unclean election check (every 5 minutes by default), while kafka-leader-election.sh acts at once on just the partitions you name.
Deleting and recreating the topic to clear the error. That loses every record in it, not just the ones an unclean election would.

Configs involved

unclean.leader.election.enable Lets a replica that is not in the ISR become leader when no in-sync replica is left, which brings the partition back online but loses the writes that replica never received (default false).
min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS.
default.replication.factor The replication factor of topics created automatically or without one (default 1); production clusters usually set 3.
broker.rack The rack or availability zone of the broker; when a topic is created, the replicas of each partition are spread across racks so one rack failing cannot take every copy.
broker.session.timeout.ms How long the registration of a KRaft broker lasts without a heartbeat to the active controller (default 9000, 9 seconds); when it runs out, the controller fences the broker and moves leadership away from it.

Reproduce it in the simulator

Sources

All troubleshooting pages