--- title: "Handle large clusters" description: "When your cluster is bigger than the Kafka Simulator renders: which topics and consumer groups are kept, and how --clamp trims brokers and partitions." url: https://monedula.dev/flock/docs/sim-link/how-to/handle-large-clusters/ --- Most clusters have more topics and consumer groups than the simulator draws. `connect` keeps the most eventful ones and says which it kept and why. Brokers and partitions over the limits fail by default, and `--clamp` trims them, the same way on every run. ## The limits A single-datacenter free-play session renders at most 10 brokers, 6 topics besides `orders`, 24 partitions per topic, 4 consumer groups and, with `--producers`, 4 producers. The [mapping reference](/flock/docs/sim-link/reference/mapping/#limits-and---clamp) has the full rules. ## Before you start - A working `connect` command for your cluster, from [Connect to a secured cluster](/flock/docs/sim-link/how-to/connect-to-a-secured-cluster/). ## Steps ### 1. Let connect pick the topics Run `connect` on the whole cluster. With more than 6 topics besides `orders`, it keeps the 6 most eventful, prints the link, and warns with what it kept, why, and what it dropped: ```text warning: selection has 10 topics besides "orders"; free play renders at most 6 extra, so it keeps the most eventful: payments (2 partitions under-replicated or offline), audit-log (lag 1840), inventory (read by 2 groups), events (5120 records), billing (412 records), clicks (idle); dropping notifications, search-index, sessions, user-profiles (name the topics you want with --topics) ``` Topics are ranked by, in order: 1. partitions under-replicated or offline: no live leader, or fewer in-sync replicas than replicas; 2. consumer lag, summed over every group that has committed offsets on the topic; 3. how many consumer groups read the topic; 4. retained records, the latest offset minus the earliest, summed over the partitions; 5. the name, alphabetically. The same cluster state keeps the same topics on every run. The warning lists at most 10 dropped names, then how many more. ### 2. Choose some topics yourself List the topics you want with `--topics` ([Choose which topics to show](/flock/docs/sim-link/how-to/choose-topics/)). Listed topics are always kept. Add `--topics-regex` to let the tool fill the free slots with the most eventful of the topics it matches: ```bash monedula-sim-link connect --brokers localhost:9092 \ --topics billing,refunds --topics-regex '.*' ``` Listing more than 6 topics besides `orders` in `--topics` fails, because the tool will not drop a topic you named without asking: ```text mapping: selection exceeds what the simulator can render (pass --clamp to trim deterministically instead): - --topics lists 7 topics besides "orders"; free play renders at most 6 extra - keeping the most eventful: payments (2 partitions under-replicated or offline), audit-log (lag 1840), inventory (read by 2 groups), events (5120 records), billing (412 records), clicks (idle); dropping sessions ``` With `--clamp`, it keeps the 6 most eventful of the listed topics instead. ### 3. Know which consumer groups are kept Only a group with a committed offset on a shown topic counts toward the 4. With more, `connect` keeps the groups with the most lag across the shown partitions, then the ones with the most members, then by name: ```text warning: 6 consumer groups have committed offsets on the rendered topics; free play renders at most 4, so it keeps the ones with the most lag: audit-sink (lag 1840), payments-ledger (lag 37), inventory-api (3 members), inventory-batch (1 member); dropping reports, search ``` A group that reads none of the shown topics is left out, and one warning lists every such group. ### 4. Trim brokers and partitions with `--clamp` Over 10 brokers, or over 24 partitions in a topic, `connect` exits 1 and prints no link. The message lists every broker, partition and `--topics` violation at once: ```text mapping: selection exceeds what the simulator can render (pass --clamp to trim deterministically instead): - topic big has 30 partitions; free play renders at most 24 per topic - keeping partitions 0..23 ``` Add `--clamp` to print the link, with a warning for each trim: ```bash monedula-sim-link connect --brokers localhost:9092 --clamp ``` ```text warning: topic big has 30 partitions; free play renders at most 24 per topic - keeping partitions 0..23 https://monedula.dev/kafka-simulator/playground?scenario=free&actions=...&size=... ``` The tool keeps the 10 lowest broker ids and partitions 0 to 23 of each topic. A replica on a broker that was trimmed away is dropped from its reassignment, with one warning per partition. ### 5. Know the other limits `--clamp` changes none of these. At most 4 members of each consumer group are shown, the first 4 the cluster reports. This limit never fails: the tool keeps the first 4 and warns, with or without `--clamp`. With `--producers`, at most 4 producers are shown. When more than 4 wrote to the shown topics in the window, the tool keeps the 4 that wrote to the most partitions, then the most recent, and warns which it kept and which it dropped. This limit never fails either. A link carries at most 2000 actions, the playground's cap, and at most 2048 characters ([Self-validation before printing](/flock/docs/sim-link/concepts/self-validation/) explains why). Over the action cap `connect` fails. Over the length it refuses the link, unless you pass `--force` to print it anyway with a warning. For either, lower `--max-records-per-partition` (default 8) or narrow the topics. ## Check it worked `connect` exits 0 and prints one warning for every limit it applied. Add `--dry-run` to see the result: the `brokers:`, `topics:` and `groups:` tables list only what was kept, and the `size:` line carries the kept counts. ## Related - [Preview a link before you share it](/flock/docs/sim-link/how-to/preview-with-dry-run/) explains the `--dry-run` output. - The [`connect` command reference](/flock/docs/sim-link/reference/cli/connect/) lists `--topics`, `--clamp`, `--max-records-per-partition` and `--force`.