Scenario 05.4.2 · No replication across AZs

Diskless vs classic Kafka: cross-AZ replication traffic

Storage · Advanced · about 3 min · Apache Kafka 4.3

diskless.enable (KIP-1163 draft) · the same 3 records go to a classic and a diskless topic across three AZs; classic replicas fetch from the leader in another AZ, diskless replicas read object storage and producers use a replica in their own AZ, and the Cross-AZ card ends at classic 7 · diskless 0 hops (one classic hop is a leader read that KIP-392 follower fetching could avoid).

What you start with

Cloud providers bill data that crosses availability zones. A classic topic with replicas in three AZs pays for it on every replica fetch, and on every produce or fetch whose leader sits in another AZ. A diskless topic (KIP-1163, Under Discussion) has no broker-to-broker replication: in the draft, replicas read the data from object storage, and producers and consumers use a replica in their own AZ. Classic Kafka can already read from a same-AZ follower (KIP-392, the PreferredReadReplica mechanism the draft reuses), but this simulator does not model that: classic reads here come from the leader, as with the default LeaderSelector. Here one producer per topic runs in az-a and one consumer group per topic in az-b; orders is classic, clicks is diskless, and each gets the same 3 records. Watch the follower lines, the object storage lane and the Cross-AZ traffic card in the metrics panel. Timings are slowed so each phase gets its own frame.

Step by step

1. Same load, two topics t+0.000s orders is a classic topic and clicks a diskless topic (diskless.enable, KIP-1163 draft), each with 1 partition, replication factor 3 and min.insync.replicas 2 on 3 brokers, one per availability zone: broker-1 in az-a, broker-2 in az-b, broker-3 in az-c. The default placement puts both partitions on all three brokers, with broker-1 (az-a) as leader. p-orders and p-clicks run in az-a; the groups g-orders and g-clicks run in az-b (client.rack). broker-1 also carries the coordinator badge. The Cross-AZ traffic card reads classic 0 · diskless 0. Open at this step: step 1, Same load, two topics
2. Both batches land in az-a t+0.200s At 150 ms both producers flushed 3 records. p-orders sent them to the orders leader, broker-1: its LEO is 3, the followers' are still 0 and the HW is 0. In the KIP-1163 draft any replica accepts a diskless produce and the client's own AZ is preferred, so p-clicks sent its 3 records to broker-1 as the az-a replica: WAL 3 under broker-1, no offsets yet. Neither produce crossed an AZ, but for orders that is luck of placement: a classic produce must go to the leader, wherever it is. Both producers are waiting; the card still reads classic 0 · diskless 0. Open at this step: step 2, Both batches land in az-a
3. Classic replicates across AZs t+0.500s At 450 ms broker-2 (az-b) and broker-3 (az-c) fetched offsets 0-2 from the leader broker-1 (az-a): two replica_fetch events, each crossing an AZ, drawn as the dashed follower lines of orders. The HW rose to 3 and p-orders was acked, 300 ms after its batch was flushed. The card now reads classic 6 · diskless 0: it counts hops, one per record a fetch moved, so each fetch of 3 records adds 3. clicks has no follower lines: its 3 records still sit in broker-1's buffer, HW 0, p-clicks waiting. Open at this step: step 3, Classic replicates across AZs
4. The classic read crosses too t+0.700s At 600 ms c-orders (g-orders, az-b) polled the 3 orders records. This simulator serves classic reads from the leader, broker-1 in az-a, so the read crossed from az-a to az-b and the card reads classic 7 · diskless 0: a consumer read counts one hop per partition read, not per record. (Real Kafka can let a consumer read from a follower in its own AZ with KIP-392: replica.selector.class=RackAwareReplicaSelector plus client.rack. The default LeaderSelector reads from the leader, as shown; the simulator does not model follower fetching, and replication would still cross AZs.) Meanwhile broker-1's buffer closed at 650 ms and its upload as wal-0001 started (600 ms upload, then a 200 ms commit): clicks HW 0, p-clicks still waiting. The upload goes to object storage, which the card does not count. Open at this step: step 4, The classic read crosses too
5. Diskless replicas read object storage t+1.750s At 1450 ms the coordinator, broker-1, committed wal-0001: offsets 0-2 for clicks, HW 3, and p-clicks was acked, 1300 ms after its batch entered the buffer. At 1750 ms broker-1, broker-2 and broker-3 each read that range from object storage into their cache (three diskless_cache_fetch, LEO 3 on every replica); no broker fetched data from another, and there is no replica_fetch for clicks (in the draft, followers still send empty fetches to the leader to track the ISR; not modelled here). In the KIP-1163 draft that is how a diskless replica gets its data. g-clicks still shows a lag of 3: c-clicks polls every 600 ms and its next poll is at 1800 ms. The card still reads classic 7 · diskless 0: object storage traffic stays in the region and is not counted. Open at this step: step 5, Diskless replicas read object storage
6. Same load, different bill t+2.400s At 1800 ms c-clicks (g-clicks, az-b) polled the 3 clicks records. In the draft a diskless fetch is served by a replica in the consumer's AZ (the PreferredReadReplica mechanism of KIP-392), so broker-2 (az-b) served it from its cache, although broker-1 in az-a leads clicks-0: no hop. Both groups have committed offset 3. The card ends at classic 7 · diskless 0. Classic: 2 replica fetches of 3 records (6 hops) plus 1 partition read from the az-a leader (1 hop); that read is the hop a classic consumer could also avoid with KIP-392 follower fetching, which this simulator does not model. Diskless: the produce went to the az-a replica, replicas read object storage, and the read came from the az-b replica. Open at this step: step 6, Same load, different bill

The takeaway

Same records, same clients, same AZs: the card ends at classic 7 · diskless 0 hops. The classic 7 are two replica fetches from the az-a leader to az-b and az-c, one hop per record, 3 each, and one consumer read from the az-a leader to the az-b group, which the card counts as one hop per partition read although it moved 3 records. One of those hops, the leader read, is not a diskless advantage: classic Kafka can serve it from the az-b follower with KIP-392 (replica.selector.class=RackAwareReplicaSelector plus client.rack), the same PreferredReadReplica mechanism the draft reuses. This simulator does not model that and reads from the leader, as the default LeaderSelector does. The savings that belong to the KIP-1163 draft alone are the other two kinds: no data replication between brokers, because replicas read object storage (no broker fetched data from another; in the draft, followers still send empty fetches to the leader to track the ISR, not modelled here), and a produce that is not tied to the leader, because any replica can take the write. The classic produce was free here only because its leader happened to sit in the producer's AZ. The upload and the cache reads still move data, to and from object storage in the same region, which the card does not count: in-region object storage traffic is typically billed per request, not as cross-AZ transfer. This is the draft model, not released Kafka behavior.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• client.rack The rack (in the cloud, the availability zone) the consumer runs in; with a broker-side replica.selector.class such as RackAwareReplicaSelector it lets the consumer fetch from a follower in its own rack instead of the leader (default empty).

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.4.2" title="Kafka Simulator: No replication across AZs" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/no-cross-az-replication/">Diskless vs classic Kafka: cross-AZ replication traffic</a></p>

Further reading

KIP-1163 on the Apache Kafka wiki

More in Storage

• Kafka retention.ms and a consumer that falls behind
• Kafka retention.bytes: size-based log retention
• Kafka OFFSET_OUT_OF_RANGE with auto.offset.reset=none
• Kafka log compaction: cleanup.policy=compact explained
• Kafka tombstones: deleting a key with delete.retention.ms
• Kafka cleanup.policy=compact,delete: both policies at once
• Kafka tiered storage: offloading old segments to remote
• Kafka tiered storage: a consumer reads from the remote tier
• Kafka local.retention.ms vs retention.ms in tiered storage
• Diskless Kafka topics (KIP-1163): the write path
• Diskless Kafka produce latency: the price of object storage

All scenarios