Scenario 07.1.4 · Throttled reassignment under load
Kafka replication throttle during partition reassignment
Reassign a partition under a replication throttle and watch the new replica catch up slowly - joining the ISR far later than an un-throttled move would.
Run this scenario in the playgroundCluster ready. t+0.000s
Four brokers, one partition, RF 3, min.ISR 2. Replica set is broker-1/2/3; broker-1 leads, broker-4 is the spare we will move data onto.
Open at this step: step 1, Cluster readySix records committed. t+4.000s
p1 has produced O1-O6 with acks=all; the high-watermark is 6 and broker-1/2/3 are all in sync. This is the data a reassignment will have to copy.
Open at this step: step 2, Six records committedThrottled reassignment begins. t+8.200s
Reassign broker-3 → broker-4 under a 1536 B/s throttle. broker-4 joins the replica set at offset 0, OUTSIDE the ISR, and the partition is marked THROTTLED while its catch-up is metered. Six records = 6144 B at 1536 B/s ⇒ ~4 s of fetching.
Open at this step: step 3, Throttled reassignment beginsPast the un-throttled window - still catching up. t+10.000s
An un-throttled move would have finished ~300 ms after the reassign (by 8300). Under the throttle broker-4 is still at offset 0, out of the ISR, so live traffic keeps the bandwidth. The simulator shows the partition under-replicated; real Kafka keeps broker-3 in the ISR until broker-4 catches up.
Open at this step: step 4, Past the un-throttled window - still catching upThrottle budget spent - replica fetches up. t+11.900s
Once ~4 s of throttled fetching elapses broker-4 catches its LEO up to the high-watermark of 6. It is fully caught up on data but the ISR-join is the next beat.
Open at this step: step 5, Throttle budget spent - replica fetches upbroker-4 joins the ISR. t+12.200s
broker-4 joins the ISR, the partition is fully replicated again, and the THROTTLED badge clears (in Kafka, broker-3 would leave the replica set only now). The move completed safely - just ~4 s later than an un-throttled reassignment would have.
Open at this step: step 6, broker-4 joins the ISRWhat you start with
A replication throttle (leader/follower.replication.throttled.rate, KIP-73) caps the catch-up replication a reassignment generates so it does not saturate the link and starve live traffic. The cost is time: the newly added replica fetches up to the high-watermark at the throttled rate, so it joins the ISR later and the move takes longer, while the old replica stays in the ISR (the simulator drops it at once and shows the partition under-replicated). We commit six records, then reassign partition 0 (broker-3 → broker-4) under a 1536 B/s throttle - six records (6144 B) take ~4 s to fetch instead of the 300 ms un-throttled window.
Step by step
- Cluster ready t+0.000s Four brokers, one partition, RF 3, min.ISR 2. Replica set is broker-1/2/3; broker-1 leads, broker-4 is the spare we will move data onto. Open at this step: step 1, Cluster ready
- Six records committed t+4.000s p1 has produced O1-O6 with acks=all; the high-watermark is 6 and broker-1/2/3 are all in sync. This is the data a reassignment will have to copy. Open at this step: step 2, Six records committed
- Throttled reassignment begins t+8.200s Reassign broker-3 → broker-4 under a 1536 B/s throttle. broker-4 joins the replica set at offset 0, OUTSIDE the ISR, and the partition is marked THROTTLED while its catch-up is metered. Six records = 6144 B at 1536 B/s ⇒ ~4 s of fetching. Open at this step: step 3, Throttled reassignment begins
- Past the un-throttled window - still catching up t+10.000s An un-throttled move would have finished ~300 ms after the reassign (by 8300). Under the throttle broker-4 is still at offset 0, out of the ISR, so live traffic keeps the bandwidth. The simulator shows the partition under-replicated; real Kafka keeps broker-3 in the ISR until broker-4 catches up. Open at this step: step 4, Past the un-throttled window - still catching up
- Throttle budget spent - replica fetches up t+11.900s Once ~4 s of throttled fetching elapses broker-4 catches its LEO up to the high-watermark of 6. It is fully caught up on data but the ISR-join is the next beat. Open at this step: step 5, Throttle budget spent - replica fetches up
- broker-4 joins the ISR t+12.200s broker-4 joins the ISR, the partition is fully replicated again, and the THROTTLED badge clears (in Kafka, broker-3 would leave the replica set only now). The move completed safely - just ~4 s later than an un-throttled reassignment would have. Open at this step: step 6, broker-4 joins the ISR
The takeaway
A replication throttle trades reassignment SPEED for live-traffic SAFETY: the new replica catches up at the configured byte rate, so the move takes longer while the old replica stays in place, but it never floods the cluster. Size the throttle to the data volume - bytes ÷ rate is roughly the catch-up time.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=07.1.4" title="Kafka Simulator: Throttled reassignment under load" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/throttled-reassignment/">Kafka replication throttle during partition reassignment</a></p>