Kafka troubleshooting
Kafka partitions are under-replicated
A partition is under-replicated when its in-sync replica set (ISR) is smaller than its replica list: at least one follower is down or has not caught up with the leader for replica.lag.time.max.ms. Find the missing broker with kafka-topics.sh --describe --under-replicated-partitions and restart or repair it; the follower rejoins the ISR on its own once it has caught up.
Reproduce it in the playground2 steps beforeTwo topics, one difference. t+0.000s
Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). Nothing has failed yet. `kafka-topics --describe` prints both topics: they share three brokers and RF 3, and they differ in min.insync.replicas, which is the difference that matters here. Then run `kafka-topics --describe --at-min-isr-partitions` on this perfectly healthy cluster and notice it is not empty.
Open at this step: step 1, Two topics, one difference1 step beforeNine records, nothing wrong. t+2.200s
The producers have written to both topics. `kafka-get-offsets --topic orders --time latest` reads the log ends without consulting any consumer group, and `kafka-topics --describe --under-replicated-partitions` answers with silence, which is the answer you want from it.
Open at this step: step 2, Nine records, nothing wrongThe moment it happensbroker-3 is killed outright. t+3.500s
One broker is gone. `kafka-topics --describe --under-replicated-partitions` now lists every partition that lost a copy, across both topics, and each row shows an ISR shorter than its replica list. `ssh broker-3` does not get that far: the machine itself is down, and the shell refuses from here without asking the cluster anything.
Open at this step: step 3, broker-3 is killed outrightRight afterThree filters, three different answers. t+5.000s
Run all three on this frame. `kafka-topics --describe --under-replicated-partitions` asks "is a copy missing", `kafka-topics --describe --under-min-isr-partitions` asks "are acks=all writes being REFUSED right now", and `kafka-topics --describe --at-min-isr-partitions` asks "is there any margin left". The same dead broker puts the two topics in different lists, and only payments is in the middle one.
Open at this step: step 4, Three filters, three different answersWhat it means
Every partition has a leader and followers that fetch from it, and the ISR is the subset that is caught up. A follower leaves the ISR when its broker loses its session with the controller (broker.session.timeout.ms, 9 seconds by default) or when it has not fetched up to the leader's log end for replica.lag.time.max.ms (30 seconds by default). In the lagging case the leader logs the shrink, for example "[Partition orders-0 broker=1] Shrinking ISR from 1,2,3 to 1,2. Leader: (highWatermark: 120, endOffset: 130). Out of sync replicas: (brokerId: 3, endOffset: 100, lastCaughtUpTimeMs: 1760000000000)." The broker metric UnderReplicatedPartitions counts the partitions that broker leads whose ISR is shorter than the replica list, so add it up across brokers. Under-replicated means redundancy is reduced, not that writes fail: with replication factor 3 and min.insync.replicas 2, an ISR of two still accepts acks=all writes. Writes are refused only once the ISR drops below min.insync.replicas, which UnderMinIsrPartitionCount and kafka-topics.sh --describe --under-min-isr-partitions report. A partition with no leader at all is offline, and no broker counts it here because no broker leads it.
Likely causes, most common first
- A broker is down or restarting. When a broker stops, crashes or loses its controller session, every partition with a replica on it shrinks its ISR at the same moment, so the count jumps. A rolling restart produces the same pattern one broker at a time, and the count should return to zero after each broker has caught up.
- A follower cannot keep up. The broker is up, but its replica fetchers fall behind: slow or failing disks, a saturated network link, a leader overloaded with produce traffic, or too few fetcher threads (num.replica.fetchers, default 1). The ISR then shrinks and expands repeatedly, which shows as non-zero IsrShrinksPerSec and IsrExpandsPerSec. Outside broker restarts both should be 0.
- A replication throttle is too low or was left on. Partition reassignments usually run under a replication throttle. If the throttled rate is below the write rate into the partition, the throttled replicas never catch up, and the throttle is only removed when someone runs kafka-reassign-partitions.sh --verify after the move.
How to fix it
- Run kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions and compare the Replicas and Isr columns. If the same broker id is missing from every row, that broker is the problem; if the missing ids vary, look at load and throttles.
- If the broker is down, bring it back. It fetches what it missed and rejoins the ISR without operator action, and IsrExpandsPerSec ticks up as it does.
- If the broker is up, check its disk latency, its network throughput and the follower metric kafka.server:type=ReplicaFetcherManager,name=MaxLag. Raising num.replica.fetchers adds fetch parallelism at the cost of CPU and memory.
- After a reassignment, run kafka-reassign-partitions.sh --verify so the throttle comes off, and raise the throttle if the follower lag is not shrinking while the move runs.
- Check UnderMinIsrPartitionCount at the same time. Under-replicated with the ISR still at or above min.insync.replicas is a repair task; under min ISR means acks=all producers are being refused right now.
- During a rolling restart, wait until the count is back to zero before restarting the next broker, so two missing replicas never stack up on the same partition.
What not to do
- Raising replica.lag.time.max.ms to quiet the alert. Slow followers then stay in the ISR, acks=all writes wait for them, and the high watermark is held back by the slowest in-sync replica.
- Reducing the replication factor to make the alert go away: the alert is about missing copies, and removing copies only hides it. Lowering min.insync.replicas does not clear it either; it only lets acks=all writes through with fewer copies.
Configs involved
replica.How long a follower may go without fetching, or without catching up to the end of the leader log, before the leader removes it from the ISR (default 30000, 30 seconds).lag. time. max. ms min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS.insync. replicas num.The fetcher threads a follower uses for each broker it replicates from (default 1); more threads help followers keep up, at the cost of CPU and memory.replica. fetchers broker.How long the registration of a KRaft broker lasts without a heartbeat to the active controller (default 9000, 9 seconds); when it runs out, the controller fences the broker and moves leadership away from it.session. timeout. ms acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
Reproduce it in the simulator
- Under-replicated vs under-min-ISR partitions in Kafka (Find the unhealthy partitions) Open at the moment it happens: Under-replicated vs under-min-ISR partitions in Kafka
- Kafka slow follower: replica.lag.time.max.ms and the ISR (Slow follower) Open at the moment it happens: Kafka slow follower: replica.lag.time.max.ms and the ISR
- Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one (Cascading failure: losing the ISR one by one) Open at the moment it happens: Kafka NOT_ENOUGH_REPLICAS: brokers failing one by one
- Graceful broker shutdown with kafka-server-stop (Stop a broker from its own shell) Open at the moment it happens: Graceful broker shutdown with kafka-server-stop
Sources
- Apache Kafka 4.3: broker config replica.lag.time.max.ms
- Apache Kafka 4.3: broker config num.replica.fetchers
- Apache Kafka 4.3: design, replication and the ISR
- Apache Kafka 4.3: monitoring (UnderReplicatedPartitions, UnderMinIsrPartitionCount, IsrShrinksPerSec, MaxLag)
- Apache Kafka 4.3: safe usage of throttled replication
- apache/kafka 4.3.0: kafka-topics.sh describe filters (TopicCommand.java)
- apache/kafka 4.3.0: Partition.isUnderReplicated counts only led partitions (Partition.scala)
- Apache Kafka 4.3.0 source: Partition.scala (Shrinking ISR from)