Monedula

Scenario 05.4.1 · Diskless: the write path

Diskless Kafka topics (KIP-1163): the write path

Storage · Advanced · about 3 min · Apache Kafka 4.3

diskless.enable (KIP-1163 draft) · records from two producers and two partitions share one WAL object in object storage; the coordinator assigns offsets at commit, the ack waits for it, and replicas read the data back from storage.

Run this scenario in the playground

Replicas cached from object storage. t+1.750s

300 ms later every replica read the committed range from object storage into its cache: LEO 2 on all three replicas of clicks-0 and 1 on all three of clicks-1, equal to the HW. The event log shows six diskless_cache_fetch events and no replica_fetch: no broker copied data from another broker. The ISR stayed at all three replicas throughout, because in the KIP-1163 draft in sync means in sync with the coordinator, not with the leader.

Open at this step: step 5, Replicas cached from object storage

What you start with

A diskless topic (KIP-1163, Under Discussion) keeps its data in object storage instead of on broker disks. In the draft a producer may send to any replica, preferably one in its own availability zone; that broker buffers records from many partitions, uploads the buffer as one WAL object, and asks the diskless coordinator (KIP-1164) to commit it. Two producers in az-a send three records to the two partitions of clicks; watch the WAL counter under broker-1, the object storage lane, the coordinator badge on broker-3 and the partition strips. Timings are slowed so each phase gets its own frame.

Step by step

  1. Diskless topic, empty cluster t+0.000s clicks is a diskless topic (diskless.enable, KIP-1163 draft) with 2 partitions on 3 brokers, one per availability zone: broker-1 in az-a, broker-2 in az-b, broker-3 in az-c. broker-1 leads clicks-0, broker-2 leads clicks-1, and broker-3 carries the coordinator badge: it plays the KIP-1164 diskless coordinator. Both producers run in az-a. Every HW and LEO is 0, the WAL counters read WAL -, and the object storage lane is empty. Open at this step: step 1, Diskless topic, empty cluster
  2. Three records buffered on broker-1 t+0.400s p1 flushed two keyless records to clicks-0 at 150 ms and p2 one record with key user-7 to clicks-1 at 250 ms. All three went to broker-1, the az-a replica, even though clicks-1 is led by broker-2: in the draft a producer may write to any replica, preferably one in its own zone. The WAL counter under broker-1 reads WAL 3 and its bar fills toward the 500 ms commit interval; the buffer already has its object key, wal-0001. No offset exists yet: every HW and LEO is still 0, and both producers are waiting. Open at this step: step 2, Three records buffered on broker-1
  3. One object uploaded, not committed t+1.300s broker-1 closed the buffer at 650 ms, 500 ms after it opened, and uploaded it as a single object, wal-0001: the dashed chip 0001 in the object storage lane (dashed means not yet committed). The upload took 600 ms and was done at 1250 ms; the commit request to the coordinator takes 200 ms more. The object holds two batches, 2 records of clicks-0 from p1 and 1 record of clicks-1 from p2, so one object mixes partitions and producers. Uncommitted batches have no offsets: every HW and LEO is still 0 and both producers are still waiting. Open at this step: step 3, One object uploaded, not committed
  4. Coordinator commits, producers acked t+1.450s At 1450 ms the coordinator on broker-3 committed wal-0001 and assigned the offsets: 0-1 in clicks-0 for p1, 0 in clicks-1 for p2. HW moved to 2 and 1. The offsets came from broker-3, not from either partition leader (broker-1, broker-2). Only now were the producers acked: p1 after 1300 ms and p2 after 1200 ms, measured from when each batch entered the buffer. In the draft the ack is the commit. Every replica's LEO is still 0, so the committed cells show grey on the partition strips: no broker has these records in its local cache yet. In this simulator that includes broker-1, which uploaded them: it too reads them back from object storage. Open at this step: step 4, Coordinator commits, producers acked
  5. Replicas cached from object storage t+1.750s 300 ms later every replica read the committed range from object storage into its cache: LEO 2 on all three replicas of clicks-0 and 1 on all three of clicks-1, equal to the HW. The event log shows six diskless_cache_fetch events and no replica_fetch: no broker copied data from another broker. The ISR stayed at all three replicas throughout, because in the KIP-1163 draft in sync means in sync with the coordinator, not with the leader. Open at this step: step 5, Replicas cached from object storage

The takeaway

One WAL object mixed two partitions and two producers. No offset existed until the coordinator on broker-3 committed the object, and neither partition leader took part. The produce ack is the commit: both producers were acked at the commit, 1300 ms and 1200 ms after their batches entered the buffer. The replicas then built their logs by reading the object back from storage, with no replica_fetch between brokers; in this simulator even broker-1, which uploaded the object, reads it back from object storage. All of this is the KIP-1163 / KIP-1164 draft model, not released Kafka behavior.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.4.1" title="Kafka Simulator: Diskless: the write path" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/diskless-write-path/">Diskless Kafka topics (KIP-1163): the write path</a></p>

Further reading

KIP-1163 on the Apache Kafka wiki

More in Storage

All scenarios