Scenario 03.1.4 · HW recompute on ISR shrink
Kafka high watermark pinned by a slow in-sync replica
HW = min(LEO) over the ISR. A lagging in-sync replica pins the HW down; removing it from the ISR makes the HW jump up.
Run this scenario in the playgroundbroker-3 throttled · still in the ISR. t+2.000s
broker-3's replica fetch is throttled, but it hasn't lagged long enough to be removed yet - the ISR is still {1,2,3}.
Open at this step: step 1, broker-3 throttled · still in the ISRRecords pile up · HW pinned at 0. t+6.100s
R1 and R2 are on brokers 1 & 2 (LEO 2), but broker-3 is stuck at LEO 0 - and it's still in the ISR. HW = min(LEO) over the ISR = 0, so NOTHING is committed or readable, even though two replicas already have the data. A slow in-sync replica holds everyone back.
Open at this step: step 2, Records pile up · HW pinned at 0broker-3 leaves the ISR · HW jumps to 3. t+8.100s
broker-3 finally lagged past the threshold and dropped out of the ISR. The HW is recomputed over the survivors {1,2}: min(3,3) = 3 - it jumps from 0 to 3 and all three records commit at once. Shrinking the ISR can ADVANCE the high watermark.
Open at this step: step 3, broker-3 leaves the ISR · HW jumps to 3What you start with
The high watermark is min(LEO) across the ISR, so a replica that is still in-sync but lagging holds the HW down even when the other replicas already have the records. broker-3 is throttled while R1-R3 pile up on brokers 1 & 2; the HW stays 0 until broker-3 drops out of the ISR.
Step by step
- broker-3 throttled · still in the ISR t+2.000s broker-3's replica fetch is throttled, but it hasn't lagged long enough to be removed yet - the ISR is still {1,2,3}. Open at this step: step 1, broker-3 throttled · still in the ISR
- Records pile up · HW pinned at 0 t+6.100s R1 and R2 are on brokers 1 & 2 (LEO 2), but broker-3 is stuck at LEO 0 - and it's still in the ISR. HW = min(LEO) over the ISR = 0, so NOTHING is committed or readable, even though two replicas already have the data. A slow in-sync replica holds everyone back. Open at this step: step 2, Records pile up · HW pinned at 0
- broker-3 leaves the ISR · HW jumps to 3 t+8.100s broker-3 finally lagged past the threshold and dropped out of the ISR. The HW is recomputed over the survivors {1,2}: min(3,3) = 3 - it jumps from 0 to 3 and all three records commit at once. Shrinking the ISR can ADVANCE the high watermark. Open at this step: step 3, broker-3 leaves the ISR · HW jumps to 3
The takeaway
Removing a lagging replica from the ISR recomputes the HW over the survivors and makes it jump up - the committed, readable position is gated by the slowest in-sync replica, not the fastest.
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.1.4" title="Kafka Simulator: HW recompute on ISR shrink" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/hw-recompute-on-isr-shrink/">Kafka high watermark pinned by a slow in-sync replica</a></p>Further reading
More in Replication
- Kafka ISR (in-sync replicas): how replicas leave and rejoin
- min.insync.replicas with acks=all: when Kafka refuses writes
- Kafka slow follower: replica.lag.time.max.ms and the ISR
- Losing two Kafka brokers with min.insync.replicas=2
- Kafka preferred leader election after a rack failure
- Kafka unclean.leader.election.enable: offline or data loss
- Kafka unclean leader election: permanent loss of acked data
- Kafka log truncation when an old leader returns
- Kafka broker.rack: rack-aware replica placement
- Kafka rack-aware reassignment after adding a rack
- Increase the replication factor of an existing Kafka topic