Monedula

Scenario 13.1.1 · Where am I: hosts, prompt, ssh

Kafka from the terminal: hosts, ssh and the KRaft controller

Terminal ops · Intermediate · about 3 min · Apache Kafka 4.3

The prompt is cluster state: ssh between brokers, meet a controller that holds no partition data, and watch ping answer from your own host rather than from the cluster.

Run this scenario in the playground

broker-3 is cut off. t+3.000s

A network partition isolated broker-3. `ping broker-3` answers from broker-2, not from the cluster - your own host reporting what it can reach from where it stands. `ssh broker-3` refuses for the same reason: you cannot log into a machine this host has no route to.

Open at this step: step 3, broker-3 is cut off

What you start with

This scenario is played from the terminal (Ctrl and the backtick key). Every Kafka tool you run is a CLIENT running on a HOST, and which host you are on decides what you can reach. A network partition isolates broker-3 for a few seconds; ping and ssh answer that out of the host you are standing on, without asking the cluster anything.

Step by step

  1. Three brokers, three racks, a KRaft quorum t+0.000s Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). You start on broker-1. Run `hostname` and `cat config/server.properties` to see which machine you are on and how it is configured. Open at this step: step 1, Three brokers, three racks, a KRaft quorum
  2. A second machine t+1.500s `ssh broker-2` moves you. Same cluster, different host: the prompt changes, and `cat config/server.properties` now reports node.id=2 and its own rack. Open at this step: step 2, A second machine
  3. broker-3 is cut off t+3.000s A network partition isolated broker-3. `ping broker-3` answers from broker-2, not from the cluster - your own host reporting what it can reach from where it stands. `ssh broker-3` refuses for the same reason: you cannot log into a machine this host has no route to. Open at this step: step 3, broker-3 is cut off
  4. Still partitioned t+5.000s `kafka-topics --describe --under-replicated-partitions` asks the CLUSTER the same question, and shows what the split cost the ISR. Then `exit` to drop back to broker-1. Open at this step: step 4, Still partitioned
  5. Healed, and a look at a controller t+7.000s The partition healed. `ssh c-1` logs into a controller voter: `ls` finds no partition directories there, because a controller's log dir holds only the cluster metadata log, which this terminal does not list. Read its config with `cat config/controller.properties`. Open at this step: step 5, Healed, and a look at a controller

The takeaway

The prompt is not decoration: it names the machine your commands run on. ssh moves you, exit comes back, and ping answers from THIS host - so the shell can tell you a machine is unreachable from here without asking a single broker, while kafka-topics asks the cluster and reports the ISR the split cost it. A controller is a machine too, but its log dir holds the metadata log, not partition data.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=13.1.1" title="Kafka Simulator: Where am I: hosts, prompt, ssh" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/terminal-hosts-and-ssh/">Kafka from the terminal: hosts, ssh and the KRaft controller</a></p>

Further reading

Apache Kafka documentation

More in Terminal ops

All scenarios