Scenario 13.2.4 · Stop a broker from its own shell
Graceful broker shutdown with kafka-server-stop
A graceful stop is not a dead machine: kafka-server-stop takes the process down, ssh still answers, the log dirs are still there, and a start brings it back.
Run this scenario in the playgroundThree replicas, min.insync.replicas 2. t+0.000s
Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). `kafka-topics --describe --topic orders` shows one partition with three replicas, all in sync, against min.insync.replicas 2 - so this topic can lose one copy and keep accepting acks=all writes. Now `ssh broker-2`: you are on the machine that is about to be taken out of service, and `hostname` is how you check that before typing anything you cannot take back.
Open at this step: step 1, Three replicas, min.insync.replicas 2Three records, everyone in sync. t+2.500s
Three records are in and every replica has them - `kafka-get-offsets --topic orders --time latest` answers from right here on broker-2, because the tools talk to the CLUSTER, not to the host you are standing on. `kafka-server-stop` is different: it takes no --bootstrap-server, only an optional --node-id or --process-role to pick among the Kafka processes on THIS machine, which is why the ssh came first. The scenario runs it on the next tick, so pressing the chip forks the sandbox onto the same action. Read the dim lines it prints.
Open at this step: step 2, Three records, everyone in syncThe process is down, the machine is not. t+3.500s
This is the distinction that matters at 3am. `exit` back to broker-1: `ping broker-2` still answers, `ssh broker-2` CONNECTS rather than refusing, `hostname` says broker-2, and `ls -lh orders-0` lists every segment file the broker had when it stopped. Kafka has already reacted - the ISR is down to two - but the host is untouched. In 13.1.4 the broker was killed the other way and ssh was refused outright; that is the difference between a stopped process and a lost machine.
Open at this step: step 3, The process is down, the machine is notWrites continue at min.insync.replicas. t+5.000s
Two more records were written while broker-2 was down. `exit` back to broker-1: `kafka-topics --describe --under-replicated-partitions` names the partition, and `kafka-get-offsets --topic orders --time latest` is two higher than it was - one copy short is still two copies in sync, which is all min.insync.replicas 2 asks for, so no acks=all write was ever refused. Then `ssh broker-2` and `ls -lh orders-0`: that log is frozen at the size it had when the process stopped, which is exactly what a stopped broker should look like. `kafka-server-start config/server.properties` brings it back, and the scenario does that on the next tick, so pressing it forks the sandbox onto the same action.
Open at this step: step 4, Writes continue at min.insync.replicasBack in service. t+8.000s
broker-2 rejoined and it caught up rather than starting from nothing: `ls -lh orders-0` here on broker-2 now carries the segment that was missing while it was down, `kafka-topics --describe --topic orders` has it back in the ISR, and `kafka-topics --describe --under-replicated-partitions` is silent again. `exit` when you are done - a stopped broker came back from a prompt you were already sitting at, which is never true of a machine that is actually gone.
Open at this step: step 5, Back in serviceWhat you start with
This scenario is played from the terminal (Ctrl and the backtick key). Every other way this simulator loses a broker is a power cut; this one is the operator doing it on purpose, from the broker's own shell, before maintenance. Here broker-2 is stopped gracefully while a producer keeps writing, the shell shows the difference between a stopped process and a lost machine, and the broker is started again from the same prompt.
Step by step
- Three replicas, min.insync.replicas 2 t+0.000s Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). `kafka-topics --describe --topic orders` shows one partition with three replicas, all in sync, against min.insync.replicas 2 - so this topic can lose one copy and keep accepting acks=all writes. Now `ssh broker-2`: you are on the machine that is about to be taken out of service, and `hostname` is how you check that before typing anything you cannot take back. Open at this step: step 1, Three replicas, min.insync.replicas 2
- Three records, everyone in sync t+2.500s Three records are in and every replica has them - `kafka-get-offsets --topic orders --time latest` answers from right here on broker-2, because the tools talk to the CLUSTER, not to the host you are standing on. `kafka-server-stop` is different: it takes no --bootstrap-server, only an optional --node-id or --process-role to pick among the Kafka processes on THIS machine, which is why the ssh came first. The scenario runs it on the next tick, so pressing the chip forks the sandbox onto the same action. Read the dim lines it prints. Open at this step: step 2, Three records, everyone in sync
- The process is down, the machine is not t+3.500s This is the distinction that matters at 3am. `exit` back to broker-1: `ping broker-2` still answers, `ssh broker-2` CONNECTS rather than refusing, `hostname` says broker-2, and `ls -lh orders-0` lists every segment file the broker had when it stopped. Kafka has already reacted - the ISR is down to two - but the host is untouched. In 13.1.4 the broker was killed the other way and ssh was refused outright; that is the difference between a stopped process and a lost machine. Open at this step: step 3, The process is down, the machine is not
- Writes continue at min.insync.replicas t+5.000s Two more records were written while broker-2 was down. `exit` back to broker-1: `kafka-topics --describe --under-replicated-partitions` names the partition, and `kafka-get-offsets --topic orders --time latest` is two higher than it was - one copy short is still two copies in sync, which is all min.insync.replicas 2 asks for, so no acks=all write was ever refused. Then `ssh broker-2` and `ls -lh orders-0`: that log is frozen at the size it had when the process stopped, which is exactly what a stopped broker should look like. `kafka-server-start config/server.properties` brings it back, and the scenario does that on the next tick, so pressing it forks the sandbox onto the same action. Open at this step: step 4, Writes continue at min.insync.replicas
- Back in service t+8.000s broker-2 rejoined and it caught up rather than starting from nothing: `ls -lh orders-0` here on broker-2 now carries the segment that was missing while it was down, `kafka-topics --describe --topic orders` has it back in the ISR, and `kafka-topics --describe --under-replicated-partitions` is silent again. `exit` when you are done - a stopped broker came back from a prompt you were already sitting at, which is never true of a machine that is actually gone. Open at this step: step 5, Back in service
The takeaway
kafka-server-stop is not a cluster command: it takes no --bootstrap-server, only an optional --node-id or --process-role, because it stops a process on the machine you are logged into - which is why you ssh to the broker first, and why running it on the wrong host is a real mistake with no warning. What it leaves behind is a stopped PROCESS on a live MACHINE, and the shell is where that difference is visible: ssh still connects, hostname still answers, and the log directories are still on disk with every segment the broker had. To the ISR it ends up like any lost replica - the partition drops to two in-sync replicas and keeps taking acks=all writes because min.insync.replicas is 2 - but a graceful stop gets there differently: controlled shutdown moves leadership off the broker and flushes its logs first. And the recovery is different in kind. A hard-killed machine refuses ssh outright and has to come back from outside; a stopped broker is one kafka-server-start away on a prompt you are already sitting at, and it rejoins the ISR by fetching only what it missed.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=13.2.4" title="Kafka Simulator: Stop a broker from its own shell" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/stop-a-broker/">Graceful broker shutdown with kafka-server-stop</a></p>Further reading
Troubleshooting
More in Terminal ops
- Kafka from the terminal: hosts, ssh and the KRaft controller
- Diagnose consumer lag with kafka-consumer-groups --describe
- Kafka log segments on disk: leader vs follower
- Under-replicated vs under-min-ISR partitions in Kafka
- Reset Kafka consumer group offsets with --to-earliest
- Set retention.ms on a live topic with kafka-configs
- Kafka partition reassignment: execute, throttle, verify
- Kafka tools on two clusters: --bootstrap-server decides