Scenario 13.1.4 · Find the unhealthy partitions

Under-replicated vs under-min-ISR partitions in Kafka

Terminal ops · Intermediate · about 3 min · Apache Kafka 4.3

Three ISR filters, three different questions: one dead broker leaves orders at min ISR with writes still flowing while payments drops below it and refuses them.

What you start with

This scenario is played from the terminal (Ctrl and the backtick key). Two topics share three brokers and differ in min.insync.replicas, which is the difference that matters here, then one broker is killed outright. The same failure is reported by three kafka-topics filters that sound alike and mean different things, and telling them apart is the difference between a page and a note in the channel.

Step by step

1. Two topics, one difference t+0.000s Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). Nothing has failed yet. `kafka-topics --describe` prints both topics: they share three brokers and RF 3, and they differ in min.insync.replicas, which is the difference that matters here. Then run `kafka-topics --describe --at-min-isr-partitions` on this perfectly healthy cluster and notice it is not empty. Open at this step: step 1, Two topics, one difference
2. Nine records, nothing wrong t+2.200s The producers have written to both topics. `kafka-get-offsets --topic orders --time latest` reads the log ends without consulting any consumer group, and `kafka-topics --describe --under-replicated-partitions` answers with silence, which is the answer you want from it. Open at this step: step 2, Nine records, nothing wrong
3. broker-3 is killed outright t+3.500s One broker is gone. `kafka-topics --describe --under-replicated-partitions` now lists every partition that lost a copy, across both topics, and each row shows an ISR shorter than its replica list. `ssh broker-3` does not get that far: the machine itself is down, and the shell refuses from here without asking the cluster anything. Open at this step: step 3, broker-3 is killed outright
4. Three filters, three different answers t+5.000s Run all three on this frame. `kafka-topics --describe --under-replicated-partitions` asks "is a copy missing", `kafka-topics --describe --under-min-isr-partitions` asks "are acks=all writes being REFUSED right now", and `kafka-topics --describe --at-min-isr-partitions` asks "is there any margin left". The same dead broker puts the two topics in different lists, and only payments is in the middle one. Open at this step: step 4, Three filters, three different answers
5. How far apart are the copies that survived? t+6.200s `kafka-replica-verification --topics-include orders` asks the question the ISR list cannot: how far apart are the copies that are still up (the real tool, deprecated since 3.9, also needs --broker-list). Here the worst spread is zero, so this failure is a MISSING copy, not a diverging one. `kafka-get-offsets --topic payments --time latest` shows the other topic sitting exactly where the kill left it, because nothing has tried to append since - and with an ISR of 2 against min.insync.replicas 3, an acks=all write would be refused if anything did. Open at this step: step 5, How far apart are the copies that survived?
6. The broker is back t+8.500s broker-3 restarted and caught up. `kafka-topics --describe --under-replicated-partitions` is silent again and `kafka-topics --describe --under-min-isr-partitions` with it, so acks=all writes to payments are accepted once more. `kafka-topics --describe --at-min-isr-partitions` is the one that still has something to say, exactly as it did before anything broke. Open at this step: step 6, The broker is back

The takeaway

Under-replicated means a copy is missing: real, but writes are still being accepted. Under min ISR means acks=all writes are being REFUSED right now, which is the one that wakes someone up. At min ISR means the partition is still serving but has no margin left - one more replica loss and it joins the second list. Run all three: orders and payments fail identically at the replica level and completely differently at the write level, and only the filter you chose tells you which. A hard-killed broker is a machine that is gone, so the shell itself refuses to connect - no Kafka tool has to answer that one.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=13.1.4" title="Kafka Simulator: Find the unhealthy partitions" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/unhealthy-partitions/">Under-replicated vs under-min-ISR partitions in Kafka</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• Kafka under-replicated partitions: causes and fixes

More in Terminal ops

• Kafka from the terminal: hosts, ssh and the KRaft controller
• Diagnose consumer lag with kafka-consumer-groups --describe
• Kafka log segments on disk: leader vs follower
• Reset Kafka consumer group offsets with --to-earliest
• Set retention.ms on a live topic with kafka-configs
• Kafka partition reassignment: execute, throttle, verify
• Graceful broker shutdown with kafka-server-stop
• Kafka tools on two clusters: --bootstrap-server decides

All scenarios