Scenario 12.0.5 · Capstone: tune a high-throughput, low-latency pipeline
Tune Kafka throughput: linger.ms, batch.size and zstd
Every dial on one MRC topic: linger.ms + batch.size + zstd compression coalesce a burst into one request, max.poll.records paces the consumer (visible lag), and the DC-B async observer trails the high-watermark then drains - throughput, latency, back-pressure, and replication lag tuned together.
Run this scenario in the playgroundTuned pipeline ready. t+0.000s
One MRC topic `events`: 2 sync replicas in DC-A + an async observer (broker-3) in DC-B. The producer is tuned for throughput (linger.ms=600, batch.size=4, compression=zstd); the consumer is back-pressured (max.poll.records=2).
Open at this step: step 1, Tuned pipeline readyBurst buffered - linger window open. t+0.800s
Six records (E1..E6) arrive in a burst. They sit in the producer buffer, not yet sent: the linger.ms=600 window is still open, and zstd shrinks each record so the EFFECTIVE batch is large enough that batch.size=4 does not trigger an early flush.
Open at this step: step 2, Burst buffered - linger window openOne compressed batch flushed - HW climbs. t+1.100s
At the linger deadline the whole burst flushes as a SINGLE compressed request (one batch_flush, not the 4+2 that batch.size=4 alone would produce). The DC-A sync ISR commits all six and the high-watermark jumps to 6 - fewer requests, higher throughput. The DC-B observer is still at offset 0.
Open at this step: step 3, One compressed batch flushed - HW climbsConsumer paced + observer lags. t+1.500s
Two dials at once. The consumer drains only 2 records per poll (max.poll.records=2) so it trails the producer - committed offset 2, position 4 of 6 (back-pressure). Meanwhile the async DC-B observer still sits at 0 while the HW is 6 (observer lag 6) - it replicates asynchronously and never counts toward acks=all.
Open at this step: step 4, Consumer paced + observer lagsDrained - consumer and observer caught up. t+2.200s
The burst is over: the consumer has worked through its 2-per-poll backlog to offset 6, and the observer has asynchronously caught up to the high-watermark (lag 0). Throughput, latency, back-pressure, and replication lag have all settled.
Open at this step: step 5, Drained - consumer and observer caught upWhat you start with
Every tuning dial on one MRC topic at once. On the producer, linger.ms + batch.size + zstd compression coalesce a six-record burst into a single time-triggered request; on the consumer, max.poll.records=2 paces the drain so lag is briefly visible; and a DC-B async observer trails the high-watermark under load. Watch how the producer, consumer, and replication knobs interact.
Step by step
- Tuned pipeline ready t+0.000s One MRC topic `events`: 2 sync replicas in DC-A + an async observer (broker-3) in DC-B. The producer is tuned for throughput (linger.ms=600, batch.size=4, compression=zstd); the consumer is back-pressured (max.poll.records=2). Open at this step: step 1, Tuned pipeline ready
- Burst buffered - linger window open t+0.800s Six records (E1..E6) arrive in a burst. They sit in the producer buffer, not yet sent: the linger.ms=600 window is still open, and zstd shrinks each record so the EFFECTIVE batch is large enough that batch.size=4 does not trigger an early flush. Open at this step: step 2, Burst buffered - linger window open
- One compressed batch flushed - HW climbs t+1.100s At the linger deadline the whole burst flushes as a SINGLE compressed request (one batch_flush, not the 4+2 that batch.size=4 alone would produce). The DC-A sync ISR commits all six and the high-watermark jumps to 6 - fewer requests, higher throughput. The DC-B observer is still at offset 0. Open at this step: step 3, One compressed batch flushed - HW climbs
- Consumer paced + observer lags t+1.500s Two dials at once. The consumer drains only 2 records per poll (max.poll.records=2) so it trails the producer - committed offset 2, position 4 of 6 (back-pressure). Meanwhile the async DC-B observer still sits at 0 while the HW is 6 (observer lag 6) - it replicates asynchronously and never counts toward acks=all. Open at this step: step 4, Consumer paced + observer lags
- Drained - consumer and observer caught up t+2.200s The burst is over: the consumer has worked through its 2-per-poll backlog to offset 6, and the observer has asynchronously caught up to the high-watermark (lag 0). Throughput, latency, back-pressure, and replication lag have all settled. Open at this step: step 5, Drained - consumer and observer caught up
The takeaway
Throughput, latency, consumer back-pressure, and cross-DC replication lag are separate dials - big compressed producer batches do not help if max.poll.records throttles the consumer or the observer cannot keep up, so you tune the whole pipeline, not one knob.
Configs in this scenario
-
linger.How long the producer waits for more records before it sends a batch that is not full (default 5 ms since Kafka 4.0, 0 before). How linger.ms worksms -
batch.The most bytes the producer puts in one batch for a partition (default 16384); a batch goes out when it is full or when linger.ms runs out, whichever comes first. How batch.size workssize -
max.The most records one poll() returns (default 500); a smaller value shortens the processing between two polls, which keeps a group member inside max.poll.interval.ms.poll. records -
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=12.0.5" title="Kafka Simulator: Capstone: tune a high-throughput, low-latency pipeline" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/tune-high-throughput-low-latency/">Tune Kafka throughput: linger.ms, batch.size and zstd</a></p>