--- title: "Kafka tools on two clusters: --bootstrap-server decides" description: "From a DR host, or via --bootstrap-server, kafka-topics --list shows the MirrorMaker 2 mirror primary.orders, not orders, and after failover g1 resumes on dr." url: https://monedula.dev/kafka-simulator/scenarios/two-clusters/ --- # Kafka tools on two clusters: --bootstrap-server decides Scenario 13.3.1 · Two clusters, two views. Terminal ops · Advanced · about 4 min · Apache Kafka 4.3. A Kafka tool talks to ONE cluster: ssh into the DR region, or point --bootstrap-server at it, and the same command answers differently. Run it: https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1 ## What you start with This scenario is played from the terminal (Ctrl and the backtick key). Two regions, one screen, and a MirrorMaker 2 link between them. Every Kafka tool is a client of the cluster its bootstrap server belongs to, so where you run it decides what you see: the same kafka-topics --list is two different answers on two different hosts, and neither of them is wrong. ## Step by step 1. **Two regions, one screen** (t+0.000s): Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). The picture above is TWO clusters with a MirrorMaker 2 link between them, but the shell opens on broker-1, which is one broker in one of them. `cluster-info` says which: it names the region you are standing in and prints that region's own KRaft quorum and brokers, not both regions' worth. Then `kafka-topics --list` for the other half of the answer. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1&t=0) 2. **The same command, on the other cluster** (t+1.100s): Three records are in and MirrorMaker 2 has carried them across the link. `ssh broker-3` moves you into the DR region and the prompt gains it. Run `cluster-info` again: a different quorum, different brokers, a different CLUSTER answering the same command. Then run `kafka-topics --list` here and read the name it gives you, because the mirror is not called orders. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1&t=1) 3. **Ask the other cluster without moving** (t+1.650s): The name is primary.orders. MirrorMaker 2 re-produces the source topic under the SOURCE cluster's alias rather than copying it by name, which is what keeps a mirror from looping back on itself, and it is why a DR runbook pinned to the bare name breaks over here: `kafka-topics --describe --topic orders` on this dr broker answers that no such topic exists. Now `exit` back to broker-1 and run `kafka-topics --list --bootstrap-server broker-3:9092`: the identical DR answer from the primary shell, because a Kafka tool follows its bootstrap server, not the host it happens to be running on. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1&t=2) 4. **The group moved with the failover** (t+1.900s): dr has been promoted. On THIS frame `kafka-consumer-groups --list` from right here on broker-1 answers with nothing at all: a group carries no region of its own, so this terminal files it under the region of the partitions it is SUBSCRIBED to, and the promotion re-subscribed g1 onto the mirrored topic. That is a rule of this simulator, not of Kafka, and it is worth knowing which: on a real primary/DR pair the group committed into the primary's own __consumer_offsets, so after its members leave it goes Empty and keeps showing up there until offsets.retention.minutes expires it, seven days by default. A group still listed on the old primary after a cutover is normal and proves nothing. Ask the cluster that owns it now with `kafka-consumer-groups --describe --group g1 --bootstrap-server broker-3:9092` and read the TOPIC and LAG columns: it resumed on primary.orders at the last translated checkpoint, which is behind its last commit on the primary. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1&t=3) 5. **Two clusters, and now they differ** (t+2.500s): One more record was produced after the failover and it landed in dr only, so the two views are no longer two spellings of the same thing. It went into the mirror topic primary.orders, a simulator shortcut: MM2 expects producers to write only local topics, so a real producer would write a dr-local orders. `kafka-get-offsets --topic orders --time latest` answers for the primary cluster and `kafka-get-offsets --topic primary.orders --time latest --bootstrap-server broker-3:9092` answers for dr; the numbers have parted. `kafka-consumer-groups --describe --group g1 --bootstrap-server broker-3:9092` shows the group caught up again on its new cluster. Whenever a Kafka tool surprises you on a multi-region setup, the first question is which cluster you just asked. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=13.3.1&t=4) ## The takeaway A Kafka tool has no idea there is a second cluster. It answers for the one its bootstrap server belongs to, which in this terminal defaults to the cluster of the host you are standing on, so the prompt is not decoration here: it is the scope of every command you type. Real Kafka 4.x tools have no default at all and make --bootstrap-server required; the host default is this simulator being kind to you. ssh into the DR region and the same kafka-topics --list names primary.orders instead of orders, because MirrorMaker 2 re-produces the source topic under the source cluster alias rather than copying it by name, and cluster-info names that region's own KRaft quorum rather than both. You do not have to move to ask, though: --bootstrap-server broker-3:9092 gets the identical answer from the primary shell, which is what you actually reach for at 3am. The consumer group is the case worth remembering, because it moves without being moved: it carries no region of its own, so this terminal files it under the region of the partitions it is subscribed to, and the failover re-subscribes it onto the mirrored topic. Here that makes it disappear from the primary cluster's group list and appear in the DR cluster's, resuming at the last translated checkpoint. Take the hand-off, not the disappearing act: the clean vanish is how this simulator keeps one group straight across two clusters, while a real primary would go on listing the group as Empty until its committed offsets expire, so a group still showing on the source side after a cutover is normal and proves nothing either way. What does not depend on the model is the divergence: from the failover on, the writes land in one cluster only, and the two views stop being two spellings of the same thing. ## Configs in this scenario - [`offsets.retention.minutes`](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_offsets.retention.minutes): How long the group coordinator keeps the committed offsets of a group with no members, or of a partition the group no longer subscribes to (default 10080, 7 days); after that the group starts again from auto.offset.reset. ## Further reading - [Apache Kafka documentation](https://kafka.apache.org/43/operations/geo-replication-cross-cluster-data-mirroring/)