Snapshot to simulator mapping

A snapshot becomes two parts of the link: the size parameter, which holds the cluster shape, and an action log that reproduces placement, failures, data volume, consumer groups and, when you ask for them, producers. Both are derived deterministically, so the same snapshot gives the same link. The playground URL format specifies the grammar.

The target is the single-datacenter free-play topology, so the link has no cluster parameter.

Pipeline and determinism

1. Brokers. The universe is the brokers in the metadata, plus any broker id that a replica, ISR or leader list refers to but the metadata lacks: a node that is dead but still assigned. They are ordered numerically when every id is an integer and lexicographically otherwise, then named broker-1 to broker-N in that order.
2. Topics. The selected topics are sorted by real name. A topic named exactly orders becomes the simulator’s fixed primary topic and goes first, and the rest are free-play extra topics in alphabetical order. Over the limit of 6 extra topics, the most eventful are kept, as described under Limits and --clamp. The DescribeConfigs result of each topic (cleanup policy, min.insync.replicas, retention) travels with it.
3. Size parameter. The brokers, partitions, replication factors and configs, by the rules below.
4. Action log. In a fixed order: remove the free-play template producer and the template consumer group, reassignments, kills, replica slowdowns, a burst of produces for each topic, consumer groups, and, only with --producers, producers. Actions are spaced 250 ms apart on the simulation timeline, starting at t=500 ms.

The size parameter

The size parameter has the form <brokers>x<partitions>x<rf>[x<extra topics>[x...]], where the base describes the orders primary topic. The tool uses the 4-segment form with extra topics, and adds a segment for the primary topic’s own min.insync.replicas, cleanup policy and retention when it carries a config override. The values are clamped to the simulator’s limits first, and written in the simulator’s own canonical form. The slug is a fixed point of its codec: decoding it with the simulator’s decodeFreePlaySize and encoding the result with encodeFreePlaySize gives the same string back. So the playground renders exactly what size says, and never rewrites the link when it loads it.

• No orders topic selected. The base still describes the primary topic, as a minimal 1x1 placeholder, because the primary topic cannot be removed through the link. A warning says so.
• Extra topics are encoded as name~parts~rf~minISR~cleanup~retMs~retBytes~~. min.insync.replicas and cleanup.policy come from the topic’s real DescribeConfigs, clamped to between 1 and the replication factor and to one of delete, compact or compact,delete, and they fall back to 1 and delete when they could not be read. retention.ms and retention.bytes are carried when they are positive; Kafka’s own -1 for disabled is treated as unset. Remote-storage tiering config (remote.storage.enable, local.retention.ms) is never fetched, so the two trailing fields stay empty.
• Primary topic config carries the same four fields for orders, but only those that differ from the simulator’s defaults, because the simulator drops a field equal to its default when it decodes the slug. Cleanup delete is left out. min.insync.replicas is left out when it equals max(1, RF-1) for the primary topic’s replication factor, which is 2 at RF 3 and 1 at RF 1 and RF 2. Retention is carried when it is set. When nothing is left, the segment is omitted entirely, and an orders on its defaults at 3 partitions and RF 3 is just 3x3x3. A min.insync.replicas of 1 at RF 3 with cleanup delete and a 7-day retention is ...xxxxxxxxxx1~~604800000~~, not ...1~d~604800000~~. In an extra topic’s segment, min.insync.replicas and the cleanup policy are always written.
• Replication factor per topic is the widest replica list seen across its partitions, clamped to the number of brokers rendered.
• Topic names with a literal x have it escaped to * in the size slug. The action log carries the name unescaped.

Limits and --clamp

The simulator renders at most:

Limit	Value
Brokers (single datacenter)	10
Partitions per topic	24
Extra topics besides orders	6
Consumer groups	4
Producers, only with --producers	4

Topics and consumer groups over their limit never fail. The tool keeps the most eventful ones, and one warning names what it kept, why, and what it dropped:

• Topics. The topics listed with --topics are kept first. The free slots go to the other selected topics, ranked by the number of partitions under-replicated or offline (no live leader, or fewer in-sync replicas than replicas), then the committed-offset lag summed over every group, then how many groups have committed offsets on the topic, then the retained records (latest minus earliest offset), then the name. The kept topics are still emitted in alphabetical order.
• Consumer groups. Only groups with a committed offset on a rendered topic count toward the 4 consumer groups. Over that, the groups with the most lag across the rendered partitions are kept, then the ones with the most members, then by id.

Brokers and partitions over their limit fail, and so does a --topics list with more than 6 topics besides orders. The message lists every broker, partition and --topics violation at once. With --clamp they are trimmed deterministically instead, with a warning for each trim: the 10 lowest broker ids, partition ids 0 to 23, and the 6 most eventful listed topics are kept. Replicas that point at brokers that were trimmed away are dropped from reassignments, with a warning for each partition. Handle large clusters walks through it.

These are the single-datacenter limits that connect targets. The MCP tool build_playground_url also validates the multi-datacenter topologies, and stretched-3 has no constant partition limit. It follows the simulator’s maxFreePlayPartitions: the largest count from 1 to 6 for which the busiest broker holds at most 3 partitions at the chosen replication factor. Partition i sits on brokers i to i+rf-1 modulo the broker count, so after p partitions a broker holds at most rf*floor(p/brokers) + min(rf, p mod brokers) of them. The result never exceeds 6: 9 brokers at RF 3 allow 6 partitions and reject 7, and 9 brokers at RF 4 allow 3.

Topic and broker naming

Simulator broker ids are positional: broker-1 to broker-N over the deterministic broker order. --dry-run prints the table of real ids to simulator ids.

A topic name has to survive two grammars: the simulator’s own sanitizer, which keeps letters, digits, ., _ and - up to 40 characters, and the action log, which reserves . as a list separator. So the tool replaces every character outside letters, digits, _ and - with -, for example billing.events becomes billing-events, and cuts the name to 40 characters. A collision after that gets a deterministic -2, -3 and so on. Every rename is warned about and listed by --dry-run.

Replica placement: RA reassignments

The simulator places every free-play topic with its default placement, a port of Apache Kafka’s replica assignment over one synthetic rack. The tool runs a Go port of the same algorithm, including its string sort of broker names, which puts broker-10 before broker-2.

For each partition, the tool computes the default placement and compares it with the real replica assignment as a set. Only a partition that differs gets an action, RA:<topic>:<partition>:<brokers>, which keeps the real preference order. A partition already on the default placement emits nothing, which keeps links short.

Approximation: the order of the replicas, which sets the preferred leader, is not enforced. The simulator elects its own leader, usually the first surviving in-sync replica, which is not always the real leader.

Failures: K kills and SR slowdowns

• A broker that is offline, meaning it is offline in the metadata or a replica list refers to it but the metadata lacks it, is added to the shape and then hard-killed with K:broker-N. The canvas shows the real dead node, and the simulator moves leaders away from it as the real cluster did.
• An under-replicated partition, whose ISR is smaller than its replica set, is approximated with SR:broker-N:1. That slows the follower replication of the lagging broker, so the simulator’s ISR-shrink logic pushes it out of the ISR too. The approximation is broker-wide: the simulator throttles all of that broker’s followers, while the real broker may lag on only some partitions. Each such broker is slowed once and warned about once.

Data volume: P produce bursts

Real record contents are never read, so data volume is shown with synthetic trailing records: searched keys, no values and simulation-clock timestamps. For each partition, the tool produces min(latest - earliest, --max-records-per-partition) records, 8 by default. That is the relative fill of a partition, not its offsets: the simulator’s offsets start at 0, and a truncated log, one whose earliest offset is above 0, only shows fewer records.

To land each record on its intended partition, the tool ports the simulator’s default keyed partitioner and searches short keys, a to z9 and then 2 to 4 characters, until every partition has enough distinct keys that hash to it.

For each topic, in this order: AP:kpN:<topic> adds a transient producer pinned to the topic, the keyed P: records follow, with partitions in ascending order and 250 ms apart, and RP:kpN removes the producer 2 seconds after the last record. At most one synthetic producer is alive at a time, because free play caps producers at 4, and the template producer p1 was removed at t=500 ms so that the playground’s auto-produce loop adds nothing on top. Without --producers that stays true for the whole session. With it, the auto-loop has the producers the tool added to run (Producers).

A partition whose rendered replicas are all dead gets no produces, because they could never be acknowledged, and a warning explains the skip.

Consumer groups: AG, AC and the lag approximation

The template group goes first. The single-datacenter free-play session starts with a consumer group c1, with one member, next to the template producer p1. The tool removes the producer with RP:p1@500 and the group with RG:c1@750 before anything else, so a fake consumer does not sit beside the real groups. It does so even when the cluster has no consumer group at all.

A real group that is also called c1 is not an exception. The template group is removed, with no warning, and the real group is added fresh later in the log by its own AG:c1:... and AC entries, so it shows its real topics and members and nothing of the template’s.

The tool lists every group with ListGroups, keeps those with the protocol type consumer, so Kafka Connect workers and share groups are skipped, describes the first 200 of them in alphabetical order with DescribeGroups, and fetches their committed offsets for the selected topics with OffsetFetch. All of these are read requests (the read-only guarantee).

The limit of 4 groups is a legibility limit of the free-play canvas, not an engine limit. For each group:

1. The topics the group has committed offsets on are intersected with the rendered topics. With no overlap, the group is skipped, and one warning lists every skipped group. Over 4 remaining groups, the ones with the most lag are kept, as described under Limits and --clamp.
2. AG:<group>:<topic1>~<topic2>... adds the group for the surviving topics. The entry has neither the optional reset code nor the optional settings field of the URL format: the snapshot keeps no group settings, and a lagging group gets its reset from the C: entry of the next step.
3. The tool sums max(0, latest offset - committed offset) over every partition the group has committed on. If the total lag is above 0, it emits C:group.<group>.consumer.autoOffsetReset:earliest right after the group is added. A fresh group defaults to latest and would start caught up; this is the one lever the action log has to make a group visibly behind.
4. Up to 4 real members join, the first 4 that the cluster reports, and a group with more members is trimmed with a warning, with or without --clamp. Each member gets an AC:<group>:<member> entry, named from the member’s client id, falling back to its member id and then to a position such as m2. Members are told apart by position, not by name, so two members that share a client id still get two distinct entries.

Approximation: the action log cannot seek a group to an offset. A lagging group starts at the log start and drains the synthetic records, so the link shows that the group is behind, not by how much.

Producers: AP additions

Off by default. Without --producers the tool sends no DescribeProducers request, and the link has no producers. With it, the tool sends one DescribeProducers request (KIP-664, Kafka 2.8 or newer) for the partitions of the selected topics, split by partition leader and bounded to 5 seconds. It is a read, but it needs READ on each topic, which is why it is opt-in (the read-only guarantee).

This is not a list of every writer. DescribeProducers reports the producer state of a broker, and a broker only has state for idempotent and transactional producers. A producer with enable.idempotence=false and no transaction has no producer id and never appears, however busy it is.

The reply lists, for each partition, the producer ids with the timestamp of each one’s last write and the first offset of its open transaction, if any. The tool then:

1. Counts a producer as active on a partition when the record timestamp of its last write there falls within --producers-window (default 15 minutes) before the snapshot time, which is taken on the machine that runs the tool, so a large clock difference to the brokers or the producers shifts the window. The record timestamp is the producing client’s clock on a CreateTime topic and the broker’s on a LogAppendTime topic, so a producer that stamps event time or replays old records can look idle. A write with no timestamp does not count. Brokers keep producer state only for producer.id.expiration.ms (24 hours by default), so a longer window shows nothing extra. Only the topics and partitions the link renders count, after the topic and partition limits, because a producer needs a topic to be pinned to.
2. Ranks the active producers by the number of partitions they wrote to, then by the most recent write, then by producer id, and keeps the top 4: free play renders at most 4 producers, and the engine ignores an add_producer past 4. Over that, one warning names the producers kept and dropped, like the limit on consumer groups.
3. Names each one pid-<producerId>. A Kafka producer id is a non-negative 64-bit integer, so the name is at most 23 characters, and it cannot clash with the template producer p1 or the transient kpN.
4. Pins it to the rendered topic it wrote to on the most active partitions. A tie goes to the earlier topic in the usual order: orders, then alphabetical.
5. Adds it after everything else, with one AP:pid-<id>:<topic> entry per producer and nothing more. The producer gets the simulator’s default config (acks=all, idempotence on), and the tool never removes it. The playground’s auto-loop, which is on by default in free play, keeps producing from it, on top of the synthetic records, so offsets keep growing.

For example, monedula-sim-link demo --producers --dry-run adds producers to its built-in fixture and ends the action list with these two entries:

  AP:pid-1001:orders@8750
  AP:pid-1002:payments@9000

In that fixture, producer 1001 wrote to all three orders partitions and to payments, so it is pinned to orders. Producer 1002 wrote to payments from inside an open transaction, which earns the warning below. A third producer wrote three hours ago, outside the default window, so it is left out.

Idempotent, not transactional. Every producer the reply lists is idempotent or transactional, and the simulator’s default producer is already idempotent. A producer with an open transaction cannot be shown as transactional: the URL grammar has no transactional.id, so the simulator would ignore a begin_transaction for it. It is added as an idempotent producer, and a warning names it and the partition. A transactional producer between transactions looks like an idempotent one, so the tool does not tell them apart.

When the read does not work, the link is still built, without the producers that could not be read, and a warning says why: a cluster that does not support the request (before Kafka 2.8, or UNSUPPORTED_VERSION), CLUSTER_AUTHORIZATION_FAILED, TOPIC_AUTHORIZATION_FAILED (the warning names the topics, and the producers of the topics that could be read are still used), another error code on some partitions, or a read that did not finish in 5 seconds. If the request works but no idempotent or transactional producer is inside the window, a warning says so, and that is not the same as nobody writing.

Self-validation

The tool verifies that its own output round-trips through the codec, and checks the final link before it prints it. Self-validation before printing describes both checks.

What is not represented

Besides the approximations above:

• Controller and KRaft state. The simulator boots its own 3-voter quorum, so the real controller id and quorum layout are ignored.
• Rack topology. The single-datacenter free-play shape has no rack axis. The real racks of the brokers are captured in the snapshot but not encoded.
• Live traffic. The snapshot is a point in time, and the link replays a static reconstruction of it.
• Producers that are not idempotent or transactional. They have no producer id on the broker, so --producers cannot see them at all.
• Producer settings and transactions. --producers shows some of who writes, not how. The broker does not report a producer’s acks, linger or batching, so every added producer has the simulator’s default config, and a transactional producer is shown as an idempotent one. At most 4 are shown, ranked by the number of partitions they wrote to.

• Diskless topics. The size format can mark a topic as diskless, from the KIP-1163 draft, but a live cluster reports no diskless.enable through DescribeConfigs, so a link from a snapshot never sets it. Only the MCP tool build_playground_url writes diskless topics.
• Client racks. The size format can give each initial producer and consumer group a client.rack, but the link creates no initial producers or consumer groups: it adds them through the action log, without a rack.