---
title: "Kafka ISR re-expansion: how a restarted broker catches up"
description: "A follower is hard-killed while O3 and O4 commit at min.insync.replicas=2. On restart it fetches the two records it missed and only then rejoins the Kafka ISR."
url: https://monedula.dev/kafka-simulator/scenarios/heal-partition-isr-reexpansion/
---

# Kafka ISR re-expansion: how a restarted broker catches up

Scenario 08.0.5 · Heal the partition: ISR re-expansion and catch-up. Chaos · Intermediate · about 3 min · Apache Kafka 4.3.

A returning broker rejoins as a follower, fetches the records it missed while down, and only after its log catches the leader is it re-added to the ISR - restoring full redundancy.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5

## What you start with

The recovery half of the chaos story. A follower is hard-killed and, while it is gone, more records commit at the min.ISR floor, so it returns stale - its log ends behind the leader’s. On restart you watch it rejoin as a follower, fetch the records it missed, and only after catching up get re-added to the ISR.

## Step by step

1. **Healthy cluster** (t+1.000s): RF=3, min.ISR=2. broker-1 leads; all three in-sync. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5&t=0)
2. **O1,O2 committed on all three** (t+5.000s): O1,O2 commit on all three (HW=2). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5&t=1)
3. **Follower killed → ISR 2** (t+8.000s): Follower broker-3 is hard-killed; the ISR shrinks to {broker-1, broker-2}. Writes continue at the floor. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5&t=2)
4. **O3,O4 committed; HW=4** (t+12.000s): O3,O4 commit on the ISR of 2 and the HW climbs to 4. broker-3 is dead and missed both - its log still ends at offset 2. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5&t=3)
5. **Restarted: caught up, ISR re-expands** (t+17.000s): broker-3 restarts as a follower and fetches the records it missed (LEO 2→4). Once its LEO reaches the high watermark, the leader asks the controller to re-add it to the ISR, restoring full RF=3 redundancy. The HW was already 4, so re-expansion restores durability headroom, not committed data. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=08.0.5&t=4)

## The takeaway

A returning replica is not trusted until it has caught up: it must fetch everything it missed before isr_expand re-adds it, restoring full redundancy without disturbing the already-advanced high watermark.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#replication)