--- title: "Graceful broker shutdown with kafka-server-stop" description: "Stop broker-2 from its own shell: ssh still connects, its log dirs stay, the Kafka ISR drops to two, acks=all writes go on, and kafka-server-start restores it." url: https://monedula.dev/kafka-simulator/scenarios/stop-a-broker/ --- # Graceful broker shutdown with kafka-server-stop Scenario 13.2.4 · Stop a broker from its own shell. Terminal ops · Intermediate · about 3 min · Apache Kafka 4.3. A graceful stop is not a dead machine: kafka-server-stop takes the process down, ssh still answers, the log dirs are still there, and a start brings it back. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4 ## What you start with This scenario is played from the terminal (Ctrl and the backtick key). Every other way this simulator loses a broker is a power cut; this one is the operator doing it on purpose, from the broker's own shell, before maintenance. Here broker-2 is stopped gracefully while a producer keeps writing, the shell shows the difference between a stopped process and a lost machine, and the broker is started again from the same prompt. ## Step by step 1. **Three replicas, min.insync.replicas 2** (t+0.000s): Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). `kafka-topics --describe --topic orders` shows one partition with three replicas, all in sync, against min.insync.replicas 2 - so this topic can lose one copy and keep accepting acks=all writes. Now `ssh broker-2`: you are on the machine that is about to be taken out of service, and `hostname` is how you check that before typing anything you cannot take back. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4&t=0) 2. **Three records, everyone in sync** (t+2.500s): Three records are in and every replica has them - `kafka-get-offsets --topic orders --time latest` answers from right here on broker-2, because the tools talk to the CLUSTER, not to the host you are standing on. `kafka-server-stop` is different: it takes no --bootstrap-server, only an optional --node-id or --process-role to pick among the Kafka processes on THIS machine, which is why the ssh came first. The scenario runs it on the next tick, so pressing the chip forks the sandbox onto the same action. Read the dim lines it prints. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4&t=1) 3. **The process is down, the machine is not** (t+3.500s): This is the distinction that matters at 3am. `exit` back to broker-1: `ping broker-2` still answers, `ssh broker-2` CONNECTS rather than refusing, `hostname` says broker-2, and `ls -lh orders-0` lists every segment file the broker had when it stopped. Kafka has already reacted - the ISR is down to two - but the host is untouched. In 13.1.4 the broker was killed the other way and ssh was refused outright; that is the difference between a stopped process and a lost machine. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4&t=2) 4. **Writes continue at min.insync.replicas** (t+5.000s): Two more records were written while broker-2 was down. `exit` back to broker-1: `kafka-topics --describe --under-replicated-partitions` names the partition, and `kafka-get-offsets --topic orders --time latest` is two higher than it was - one copy short is still two copies in sync, which is all min.insync.replicas 2 asks for, so no acks=all write was ever refused. Then `ssh broker-2` and `ls -lh orders-0`: that log is frozen at the size it had when the process stopped, which is exactly what a stopped broker should look like. `kafka-server-start config/server.properties` brings it back, and the scenario does that on the next tick, so pressing it forks the sandbox onto the same action. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4&t=3) 5. **Back in service** (t+8.000s): broker-2 rejoined and it caught up rather than starting from nothing: `ls -lh orders-0` here on broker-2 now carries the segment that was missing while it was down, `kafka-topics --describe --topic orders` has it back in the ISR, and `kafka-topics --describe --under-replicated-partitions` is silent again. `exit` when you are done - a stopped broker came back from a prompt you were already sitting at, which is never true of a machine that is actually gone. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.2.4&t=4) ## The takeaway kafka-server-stop is not a cluster command: it takes no --bootstrap-server, only an optional --node-id or --process-role, because it stops a process on the machine you are logged into - which is why you ssh to the broker first, and why running it on the wrong host is a real mistake with no warning. What it leaves behind is a stopped PROCESS on a live MACHINE, and the shell is where that difference is visible: ssh still connects, hostname still answers, and the log directories are still on disk with every segment the broker had. To the ISR it ends up like any lost replica - the partition drops to two in-sync replicas and keeps taking acks=all writes because min.insync.replicas is 2 - but a graceful stop gets there differently: controlled shutdown moves leadership off the broker and flushes its logs first. And the recovery is different in kind. A hard-killed machine refuses ssh outright and has to come back from outside; a stopped broker is one kafka-server-start away on a prompt you are already sitting at, and it rejoins the ISR by fetching only what it missed. ## Configs in this scenario - [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/) - [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/) ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_controlled.shutdown.enable) ## Troubleshooting - [Kafka under-replicated partitions: causes and fixes](https://monedula.dev/kafka-simulator/troubleshooting/under-replicated-partitions/)