--- title: "Fidelity" description: "What the Kafka Simulator models, what it approximates and what it leaves out, against its modeled baseline of Apache Kafka 4.3." url: https://monedula.dev/flock/docs/kafka-simulator/concepts/fidelity/ --- The [Kafka Simulator page](/kafka-simulator/) shows the first and last list and links to the middle one; all three follow the release you are using. Modeled baseline: Apache Kafka 4.3. Not for production-as-is. The playground says the same in the **⋯** menu of its top bar: "modeled baseline · 4.3". ## What it models - Producer → leader → follower fetch → HW/LEO advance, across multiple topics - Producer tuning - acks, idempotence & `max.in.flight` ordering, compression (zstd/gzip/snappy), `buffer.memory` backpressure, byte-rate quotas - Transactions & EOS - PIDs, sequences, LSO, zombie fencing - ISR membership, leader epoch, unclean leader election, throttled reassignment (`replication.throttled.rate`) - Retention - time- & size-based (`retention.bytes`), compaction & tombstones, tiered storage with `local.retention.ms` (KIP-405) - Consumer groups (eager & cooperative rebalances), static membership (KIP-345), share groups (KIP-932) - Control plane - KRaft quorum, controller epoch & failover, plus the pre-4.0 ZooKeeper control plane for comparison (Kafka 4.x has no ZooKeeper mode) - Multi-DC DR - stretched 2.5-DC / MRC, observer promote & demote, active-passive failover & failback, active-active; MirrorMaker 2 offset translation & checkpoints; Cluster Linking (offset-preserving mirror, promote, bidirectional switchover) - Schema Registry - subjects, versions, compatibility modes - Authorization - ACLs & RBAC (principal / resource / operation, role bindings, deny-by-default) - Hard kill, graceful shutdown, restart, slow follower, network partition - Real Kafka metric names, directionally correct The observers of the **2.5-DC stretched** sandbox come from Confluent's Multi-Region Clusters, a Confluent Platform feature rather than part of Apache Kafka. Cluster Linking, one of the two ways to mirror between regions, is a Confluent Platform feature too. ## What it approximates Some things are modeled by their effect, and some behave differently from Kafka 4.3 on purpose, so that a scenario stays short and readable. Most lines say what the simulator does and what Kafka does instead, a few describe an effect the simulator only approximates, and the scenarios point it out where it matters. ### Producer - `batch.size` - counted in records here; Kafka counts bytes (default 16384) - Keyless sticky partitioning - the simulator moves to the next partition after each batch; Kafka 4.x moves after `batch.size` bytes, to a random partition weighted toward brokers with shorter queues - Client configs fixed at creation - `acks`, `compression.type` and `max.in.flight.requests.per.connection` on a producer, and `partition.assignment.strategy` on a group, change while running here; in Kafka they are fixed when the client is created, so you recreate the producer or restart the consumers - Sends to a leaderless partition - fail at once here (acks=0 drops the record and reports it sent, acks=1/all give up after their retries); a Kafka producer holds the batch until a leader appears, or until `delivery.timeout.ms` (default 120 s) expires it with a TimeoutException - Byte-rate quotas & compression - modeled by their effect on batch sizes and timing, not through real codecs or wall-clock byte rates ### Consumer groups - Auto-commit timing - offsets commit on every next poll; Kafka commits in poll() once `auto.commit.interval.ms` (default 5 s) has passed, so a crashed consumer can replay more records - Group protocol label - a cooperative-sticky group is shown on the consumer protocol (KIP-848); in Kafka cooperative-sticky is an assignor of the classic protocol (KIP-429), and on the consumer protocol the broker assigns partitions and `partition.assignment.strategy` is not used ### Transactions - Transaction markers - commit and abort markers take no offset here and land on every partition in one step; in Kafka each is a control record with an offset of its own, written partition by partition, so a read_committed consumer can see a commit on one partition before another - Transaction timeout - starts at `begin_transaction` here, fires exactly at the deadline and shows a `transaction_timed_out` error; Kafka counts from the first partition added, aborts on a periodic coordinator check (every 10 s by default), and the next send or commit from the producer fails with an epoch error - Zombie fencing - a fenced producer fails at `begin_transaction` here, and its epoch stays put across commits; in Kafka `beginTransaction()` is local, so the fence lands on the next send or commit, and `transaction.version` 2 bumps the epoch at every commit. Consumer-group fencing of transactional offset commits (KIP-447) is not modeled ### Storage - Log segments - retention, compaction and tiering act on single records or batches and can reach the newest data at once; Kafka acts on whole segments, never compacts the active segment, and rolls it before deleting or tiering it - Terminal disk view - a teaching scale: a record counts as 1 KiB and a segment rolls every 4 records; Kafka sizes records in bytes and rolls a segment at `segment.bytes` (default 1 GiB) or `segment.ms` (default 7 days) ### Replication & operations - Clean-election leader choice - the simulator elects the in-sync replica with the highest log end offset; Kafka elects the first live in-sync replica in assignment order, so the two can choose different leaders - Network partitions - modeled at the offset level (on heal a diverged replica truncates back to the high watermark; Kafka truncates where its leader epoch diverges, which can sit above it), not packet loss or latency jitter - Leadership in a network split - where no controller quorum is modeled, or each region has its own, leadership moves to the side holding most of the replicas; the Kafka controller fences the brokers it cannot reach and elects a leader from the ISR it can reach - Reassignment hand-over - removed replicas, the leader included, leave as soon as a reassignment runs; Kafka keeps them in the replica set and ISR until the added replicas catch up - Replication throttles - `replication.throttled.rate` acts on catch-up speed in simulated time, not on real byte rates - Turning on `unclean.leader.election.enable` - elects an out-of-sync leader at once here; the KRaft controller acts on its next periodic check, or when you run `kafka-leader-election --election-type unclean` - `auto.leader.rebalance.enable` - off in scripted runs, and in free play it returns leadership as soon as the preferred replica rejoins the ISR; Kafka defaults it on and checks every 300 s (`leader.imbalance.check.interval.seconds`) ### Control plane - KRaft controller catch-up - a newly elected controller can be behind and visibly replays the metadata log before it serves; Raft elects only a voter holding every committed record, which serves once it commits a record in its new epoch - ZooKeeper controller hand-over - when the controller broker dies, the simulator gives the role to the lowest-id live broker; in Kafka the first broker to recreate the ephemeral `/controller` znode wins - KRaft leadership after a heal - the lowest-id voter takes the active controller role back; Raft has no preferred leader, so the sitting controller keeps it and the returning voter rejoins as a follower ### Multi-DC - MRC observer demotion - in scripted runs a promoted observer stays a synchronous replica when the sync replicas return (free-play MRC demotes it on recovery by default); Confluent demotes it automatically - MirrorMaker 2 failback - modeled as a reverse re-sync, and producers can write to the mirror topic after a failover; MirrorMaker 2 has no failback action, and producers write to local topics - MirrorMaker 2 offset translation - exact, record by record, and a group committed past the mirror resumes at the DR high watermark; MirrorMaker 2 keeps sparse offset syncs, so a translated offset can land a little early, and a group resumes from its last translated checkpoint - Cluster Linking details - mirror topics carry a source prefix (`primary.orders`), and promote removes the whole link even with the source unreachable or lagging; Confluent keeps the source topic name by default, and promote stops one mirror topic and needs the source reachable with no lag (failover otherwise) ### Timing - Event timing - events are discrete and seeded: their relative order is modeled, not exact latency or throughput - Compressed timeouts - timings are compressed so a scenario plays in seconds, and some values, such as consumer session timeouts, sit below limits a broker enforces (`group.min.session.timeout.ms`) ## What it does not model - GC pauses, OS page cache, network buffers, byte-exact sizes - SASL, mTLS, encrypted transport - the auth handshake & wire crypto (ACL/RBAC authorization itself is modeled) - Kafka Streams & Connect - core broker/client protocol only - Magnitudes of metric movement - calibrated direction, illustrative scale The **Help** button in the playground's top bar shows the same simplifications, with more detail on what is modeled and not modeled, next to the keyboard shortcuts. ## Related - [How the simulator works](/flock/docs/kafka-simulator/concepts/how-the-simulator-works/) explains the engine behind these lists. - [Report an issue](/flock/docs/kafka-simulator/how-to/report-an-issue/) when the simulator does something these pages do not explain.