Scenario 08.0.5 · Heal the partition: ISR re-expansion and catch-up
Kafka ISR re-expansion: how a restarted broker catches up
A returning broker rejoins as a follower, fetches the records it missed while down, and only after its log catches the leader is it re-added to the ISR - restoring full redundancy.
Run this scenario in the playgroundHealthy cluster. t+1.000s
RF=3, min.ISR=2. broker-1 leads; all three in-sync.
Open at this step: step 1, Healthy clusterO1,O2 committed on all three. t+5.000s
O1,O2 commit on all three (HW=2).
Open at this step: step 2, O1,O2 committed on all threeFollower killed → ISR 2. t+8.000s
Follower broker-3 is hard-killed; the ISR shrinks to {broker-1, broker-2}. Writes continue at the floor.
Open at this step: step 3, Follower killed → ISR 2O3,O4 committed; HW=4. t+12.000s
O3,O4 commit on the ISR of 2 and the HW climbs to 4. broker-3 is dead and missed both - its log still ends at offset 2.
Open at this step: step 4, O3,O4 committed; HW=4Restarted: caught up, ISR re-expands. t+17.000s
broker-3 restarts as a follower and fetches the records it missed (LEO 2→4). Once its LEO reaches the high watermark, the leader asks the controller to re-add it to the ISR, restoring full RF=3 redundancy. The HW was already 4, so re-expansion restores durability headroom, not committed data.
Open at this step: step 5, Restarted: caught up, ISR re-expandsWhat you start with
The recovery half of the chaos story. A follower is hard-killed and, while it is gone, more records commit at the min.ISR floor, so it returns stale - its log ends behind the leader’s. On restart you watch it rejoin as a follower, fetch the records it missed, and only after catching up get re-added to the ISR.
Step by step
- Healthy cluster t+1.000s RF=3, min.ISR=2. broker-1 leads; all three in-sync. Open at this step: step 1, Healthy cluster
- O1,O2 committed on all three t+5.000s O1,O2 commit on all three (HW=2). Open at this step: step 2, O1,O2 committed on all three
- Follower killed → ISR 2 t+8.000s Follower broker-3 is hard-killed; the ISR shrinks to {broker-1, broker-2}. Writes continue at the floor. Open at this step: step 3, Follower killed → ISR 2
- O3,O4 committed; HW=4 t+12.000s O3,O4 commit on the ISR of 2 and the HW climbs to 4. broker-3 is dead and missed both - its log still ends at offset 2. Open at this step: step 4, O3,O4 committed; HW=4
- Restarted: caught up, ISR re-expands t+17.000s broker-3 restarts as a follower and fetches the records it missed (LEO 2→4). Once its LEO reaches the high watermark, the leader asks the controller to re-add it to the ISR, restoring full RF=3 redundancy. The HW was already 4, so re-expansion restores durability headroom, not committed data. Open at this step: step 5, Restarted: caught up, ISR re-expands
The takeaway
A returning replica is not trusted until it has caught up: it must fetch everything it missed before isr_expand re-adds it, restoring full redundancy without disturbing the already-advanced high watermark.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=08.0.5" title="Kafka Simulator: Heal the partition: ISR re-expansion and catch-up" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/heal-partition-isr-reexpansion/">Kafka ISR re-expansion: how a restarted broker catches up</a></p>