Monedula

Scenario 05.4.3 · The price is latency

Diskless Kafka produce latency: the price of object storage

Storage · Advanced · about 3 min · Apache Kafka 4.3

diskless.enable (KIP-1163 draft) · in this simulator, the same acks=all produce is acked in 300 ms on a classic topic and 1300 ms on a diskless one (commit interval + upload + commit); a burst that fills the buffer is acked in 800 ms, because a full buffer closes without waiting for the timer.

Run this scenario in the playground

Size beat the timer: 800 ms. t+3.150s

At 2850 ms the coordinator committed wal-0002 (offsets 1-4, HW 5) and p-clicks was acked: "Producer p-clicks acked in 800 ms", the 600 ms upload plus the 200 ms commit, with no time spent waiting for the timer. At 3150 ms every replica cached the new records (LEO 5 on all three). The timer due at 2550 ms found no open buffer and did nothing. Acks in this run: 300 ms classic, 1300 ms diskless under light load, 800 ms diskless when the buffer filled up.

Open at this step: step 7, Size beat the timer: 800 ms

What you start with

A diskless topic (KIP-1163, Under Discussion) saves the replication traffic between brokers (05.4.2), and pays in latency: in the draft a produce is acked only after the broker's buffer has closed, been uploaded to object storage as one object and been committed by the coordinator. The buffer closes after the commit interval or when it is full, whichever comes first. Here p-orders writes to the classic topic orders and p-clicks to the diskless topic clicks, both with acks=all from az-a: first one record each, then a burst of 4 records to clicks. Watch the "acked in" lines in the event log, the WAL counter under broker-1 and the object storage lane. Timings are slowed so each phase gets its own frame, and the buffer limit is 4 records, standing in for the draft's byte limit.

Step by step

  1. Two topics, one clock t+0.000s orders is a classic topic and clicks a diskless topic (diskless.enable, KIP-1163 draft), each with 1 partition and replication factor 3 on 3 brokers, one per availability zone: broker-1 in az-a, broker-2 in az-b, broker-3 in az-c. broker-1 leads both partitions and carries the coordinator badge. p-orders and p-clicks both run in az-a with acks=all. The clicks buffer closes after a 500 ms commit interval or at 4 records, whichever comes first; the upload then takes 600 ms and the commit 200 ms. These are sandbox timings, slowed so each phase gets its own frame. Every HW and LEO is 0. Open at this step: step 1, Two topics, one clock
  2. One record each, both on broker-1 t+0.200s At 150 ms each producer flushed one record. orders: broker-1, the leader, appended it at offset 0; the followers have not fetched it yet (LEO 1 / 0 / 0, HW 0). clicks: the record went into broker-1's WAL buffer (WAL 1, object key wal-0001) and has no offset yet. One record is far from the 4-record limit, so only the 500 ms timer will close this buffer, at 650 ms. Both producers are waiting. Open at this step: step 2, One record each, both on broker-1
  3. Classic acked in 300 ms t+0.500s At 450 ms broker-2 and broker-3 fetched offset 0 from broker-1, the orders HW moved to 1 and p-orders was acked: the event log reads "Producer p-orders acked in 300 ms", measured from the flush at 150 ms. Those 300 ms are this simulator's replica fetch delay, not a network measurement: a classic acks=all produce waits for its in-sync followers, which in a real cluster usually takes a few to tens of milliseconds. Meanwhile the clicks record is still in the buffer (WAL 1, HW 0). Open at this step: step 3, Classic acked in 300 ms
  4. The timer closed a one-record object t+0.700s At 650 ms, 500 ms after it opened, the commit interval closed the buffer with a single record in it, and broker-1 started uploading it as wal-0001 (the dashed chip in the object storage lane: not committed yet). The upload takes 600 ms and the commit request 200 ms more, so this record cannot be acked before 1450 ms. The clicks HW is still 0 and p-clicks is waiting. Open at this step: step 4, The timer closed a one-record object
  5. Diskless acked in 1300 ms t+1.750s At 1450 ms the coordinator committed wal-0001 and assigned offset 0 (HW 1), and only then was p-clicks acked: "Producer p-clicks acked in 1300 ms", measured from when its batch entered the buffer at 150 ms. That is 500 ms waiting for the timer, 600 ms of upload and 200 ms of commit. At 1750 ms every replica read the record from object storage into its cache (LEO 1 on all three). Same zone, same acks=all: 300 ms for classic, 1300 ms for diskless. Open at this step: step 5, Diskless acked in 1300 ms
  6. A full buffer closes at once t+2.100s At 2050 ms p-clicks flushed a burst of 4 records in one batch. All 4 went into a new buffer on broker-1, the 4th append reached the 4-record limit, and the buffer closed in the same millisecond: broker-1 started uploading wal-0002 at 2050 ms instead of waiting for the timer, which was due at 2550 ms. One object now carries 4 records, where phase 1 spent a whole object on 1. The clicks HW is still 1 and p-clicks is waiting for the commit. Open at this step: step 6, A full buffer closes at once
  7. Size beat the timer: 800 ms t+3.150s At 2850 ms the coordinator committed wal-0002 (offsets 1-4, HW 5) and p-clicks was acked: "Producer p-clicks acked in 800 ms", the 600 ms upload plus the 200 ms commit, with no time spent waiting for the timer. At 3150 ms every replica cached the new records (LEO 5 on all three). The timer due at 2550 ms found no open buffer and did nothing. Acks in this run: 300 ms classic, 1300 ms diskless under light load, 800 ms diskless when the buffer filled up. Open at this step: step 7, Size beat the timer: 800 ms

The takeaway

Three acks, one lesson: 300 ms for classic, 1300 ms for a lone diskless record and 800 ms for a diskless burst that filled the buffer. A diskless ack waits for three things in a row: the buffer closing, the upload, and the commit at the coordinator. The buffer closes on time or on size, whichever comes first, so a quiet broker waits out the whole interval and a busy one closes early. The KIP-1163 draft sets these as diskless.append.commit.interval.ms and diskless.append.buffer.max.bytes, with targets of 250 ms and 4 MiB, and aims for a produce latency around 500 ms at P50 and 1-2 s at P99, with estimates of about 100 ms (P50) to 200-400 ms (P99) for the upload and about 10 ms (P50) to 20-50 ms (P99) for the commit. This run used 500 ms, 4 records, 600 ms and 200 ms instead, scaled up so each step is visible: the numbers are illustrative, the shape is the lesson. A longer interval or a larger buffer means fewer, bigger objects, and object storage typically bills per request, so fewer uploads cost less; the price is a longer wait for every record in the buffer. Phase 1 spent a whole object on one record, the burst put four records in one. The classic 300 ms is this simulator's replica fetch delay; a real acks=all produce in one region usually takes a few to tens of milliseconds, so in relative terms the real gap is wider than it looks here. All of this is the draft model, not released Kafka behavior.

Configs in this scenario

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=05.4.3" title="Kafka Simulator: The price is latency" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/diskless-produce-latency/">Diskless Kafka produce latency: the price of object storage</a></p>

Further reading

KIP-1163 on the Apache Kafka wiki

More in Storage

All scenarios