Scenario 07.1.2 · Scale in: drain and remove a broker

Remove a Kafka broker safely: drain it first

Operations · Intermediate · about 3 min · Apache Kafka 4.3

Reassign a broker’s partitions away first, let the replacements join the ISR, then remove the now-empty broker cleanly.

What you start with

Removing a broker that still hosts replicas is disruptive - every partition it holds drops it from the ISR, and a re-election fires for any it was leading. The safe way to scale in is to DRAIN first: reassign every partition off the broker, let the replacements catch up into the ISR, then remove the now-empty broker. broker-4 hosts replicas on partitions 1 and 2, so we reassign both away before removing it.

Step by step

1. Cluster ready t+0.000s Four brokers, RF 3, min.ISR 2. broker-4 is a replica on partitions 1 and 2 (it is the broker we will scale in). Open at this step: step 1, Cluster ready
2. Records produced t+6.000s O5 committed to partition 1, O4 to partition 2 (acks=all). High-watermark 1 on both - so a drained replacement will have real data to catch up on. Open at this step: step 2, Records produced
3. Draining partition 1 t+8.050s reassign_partition moves partition 1 off broker-4: broker-1 joins at offset 0 and starts fetching, and leader broker-2 is untouched. The simulator drops broker-4 at once; real Kafka keeps it in the replica set and ISR until broker-1 has caught up. Open at this step: step 3, Draining partition 1
4. Partition 1 drained t+8.500s broker-1 caught up to the high-watermark and joined the ISR. Partition 1 no longer references broker-4. Open at this step: step 4, Partition 1 drained
5. Draining partition 2 t+12.050s reassign_partition moves partition 2 off broker-4: broker-2 joins at offset 0 and fetches. Leader broker-3 stays. broker-4 now hosts no replicas in the simulator; in Kafka it would keep this one until broker-2 joins the ISR. Open at this step: step 5, Draining partition 2
6. Partition 2 drained t+12.500s broker-2 is in the ISR. broker-4 is fully drained - it leads nothing and replicates nothing. Open at this step: step 6, Partition 2 drained
7. broker-4 removed t+16.050s remove_broker on the empty broker is a clean no-op: no ISR shrink, no leader election, nothing under-replicated - because the drain already moved every replica away. Open at this step: step 7, broker-4 removed

The takeaway

Always drain a broker before removing it: once it holds nothing, remove_broker is a clean no-op - no ISR shrink, no leader election, nothing left under-replicated.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=07.1.2" title="Kafka Simulator: Scale in: drain and remove a broker" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/scale-in-drain-broker/">Remove a Kafka broker safely: drain it first</a></p>

Further reading

Apache Kafka documentation

More in Operations

• Add a Kafka broker and reassign partitions onto it
• kafka-reassign-partitions: change a replica set by hand
• Kafka replication throttle during partition reassignment
• Change Kafka min.insync.replicas at runtime, no restart
• producer_byte_rate: how a Kafka client quota throttles

All scenarios