--- title: "Kafka partition reassignment: execute, throttle, verify" description: "Move orders-0 onto an empty broker-4 with kafka-reassign-partitions under a throttle, confirm with --verify, then run a preferred leader election to finish." url: https://monedula.dev/kafka-simulator/scenarios/move-a-partition/ --- # Kafka partition reassignment: execute, throttle, verify Scenario 13.2.3 · Move a partition: execute, throttle, verify. Terminal ops · Intermediate · about 4 min · Apache Kafka 4.3. A planned move from the shell: execute a reassignment JSON under a throttle, watch the new replica catch up, verify it, and put leadership where the plan wants it. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3 ## What you start with This scenario is played from the terminal (Ctrl and the backtick key). Moving a partition is the operation every Kafka operator eventually has to do - draining a broker, filling a new one - and it is the one that most often gets abandoned half-done. Here orders-0 is moved off broker-1 onto an empty broker-4 under a replication throttle, the move is watched while it runs, verified when it lands, and then leadership is put back on the replica the plan made preferred. ## Step by step 1. **Three copies on three brokers, a fourth standing empty** (t+0.000s): Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). `kafka-topics --describe --topic orders` shows where orders-0 lives right now: three replicas, three in sync, and no broker 4 anywhere in the row. `ls` here on broker-1 lists the partition directory from the other side, off the disk rather than off the metadata. The fourth machine is real and it is empty - `ssh broker-4`, `ls`, `exit` - and putting one partition on it is the whole scenario. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=0) 2. **Eight records, nothing moving yet** (t+2.500s): Eight records are in and all three copies have them. `kafka-get-offsets --topic orders --time latest` reads the end of the log straight from the brokers, with no consumer group involved, and `kafka-topics --describe --topic orders` still shows the original three replicas with a full ISR. `kafka-reassign-partitions --list` is the baseline for everything that follows: a cluster with nothing in flight says so in one line. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=1) 3. **The move is running, and it returned immediately** (t+3.500s): This is the command that started the move, and pressing it forks the sandbox onto the same two actions the scenario fired here: `kafka-reassign-partitions --execute --reassignment-json '{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[4,2,3]}]}' --throttle 1000000`. A real cluster takes the plan as a file, --reassignment-json-file plan.json; this terminal has no files to write, so it accepts the JSON inline. Note the broker NUMBERS in the JSON, and that it came back instantly - the replica set is already rewritten, so `kafka-topics --describe --topic orders` names broker 4 in Replicas with an ISR one short (in the simulator; real Kafka lists 4,2,3,1 with broker-1 still in the ISR until the copy finishes) and spells the move out in AddingReplicas and RemovingReplicas, and `kafka-reassign-partitions --list` names the partition too. None of the DATA has moved yet. Read the warning the execute printed: the next two frames are about honouring it. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=2) 4. **The new copy is behind, the partition still serves** (t+6.000s): Two more records landed while the move was running and the partition took them - `kafka-get-offsets --topic orders --time latest` is higher than it was, because two in-sync replicas still meet min.insync.replicas 2 and an acks=all write needs no more than that (real Kafka keeps the outgoing broker-1 in the ISR, and leading, until the move completes, so three would be in sync). The new copy is another matter: `ssh broker-4`, then `ls -lh orders-0`, and `exit`. The directory is there and holds nothing yet, and `kafka-reassign-partitions --verify --reassignment-json '{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[4,2,3]}]}'` puts the same fact in one line (inline JSON again, standing in for --reassignment-json-file). One honest aside while you wait, because TWO throttles are in play and only one of them is why this is taking seconds. The bytes-per-second you passed to the execute is recorded on the CLUSTER and is what the verify will clear, but this simulator buckets it onto a four-step control that only the cross-region mirror reads - on a single-region cluster like this one it changes no timing at all. The wait you are watching comes from the SCENARIO's own leader/follower.replication.throttled.rate, which no command can move. In free play that rate is the replication throttle timing field, off by default, so the same move there finishes instantly unless you set it. On a real cluster the flag is what slows it down. `kafka-reassign-partitions --help` carries the whole divergence. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=3) 5. **Verified, and the throttle comes off with it** (t+7.500s): broker 4 is in the ISR, so the move is finished - and `kafka-reassign-partitions --verify --reassignment-json '{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[4,2,3]}]}'` is how you find that out (inline JSON again, standing in for --reassignment-json-file). It does one more thing while it is there: with nothing still in progress it CLEARS the throttle the execute set, which is exactly what that warning was asking for. The scenario does the same on the next tick, so pressing the chip forks the sandbox onto the same action. `kafka-reassign-partitions --list` has nothing left to report, and `ssh broker-4`, `ls -lh orders-0`, `exit` shows the segment files that were not there on the previous frame. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=4) 6. **Moved, and still led from the wrong broker** (t+9.000s): The throttle is off and the data is in place. `kafka-topics --describe --topic orders` shows broker 4 FIRST in Replicas, the preferred leader, but not the Leader, because this simulator handed leadership to broker-2 when broker-1 left the list. Real Kafka keeps broker-1 leading until the move completes, then elects broker 4, the first in-sync target replica, so there this gap only appears when the old leader stays in the list. `kafka-leader-election --election-type preferred --all-topic-partitions` closes it; the scenario runs it on the next tick, so pressing it forks the sandbox onto the same action. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=5) 7. **Leadership where the plan put it** (t+10.500s): `kafka-topics --describe --topic orders` now reads Leader 4 against Replicas 4,2,3 with all three in sync: the partition is led by the replica the plan made preferred, and it only went there because broker 4 was already in the ISR - an out-of-sync preferred replica is skipped. The other end of the move is on disk: `ls` here on broker-1 has nothing left to list, which is what draining a broker is for. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.3&t=6) ## The takeaway A reassignment is a plan, a wait, and two things you have to finish yourself. --execute returns the moment the replica set is rewritten, so success there means the move STARTED: the new replica is in the assignment at offset zero and outside the ISR, which is what --list and the Adding/Removing annotation on --describe are telling you while the catch-up runs. --verify is the only thing that tells you it landed, and it is not merely a read: once every named partition is complete it clears the throttle the execute put on, which is exactly what the warning printed at execute time is asking you to do. And a move does not by itself put the preferred replica in charge: the broker you put first in the replica list is the PREFERRED leader, and a current leader that stays in the list keeps leading. Real Kafka only elects a new leader, the first in-sync target replica, when the move drops the current one; this simulator moves leadership at execute time instead, which is why the move in this run lands led from broker-2. kafka-leader-election --election-type preferred is the step that closes that gap, and it only promotes a preferred replica that is already in the ISR. ## Configs in this scenario - [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/) - [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/) ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/operations/basic-kafka-operations/)