Scenario 01.2.3 · Compression and batch shape

Kafka compression.type: fewer, larger producer batches

Producers & EOS · Intermediate · about 3 min · Apache Kafka 4.3

compression.type packs more records into the same batch.size budget: the same record stream flushes as fewer, larger batches with fewer produce requests.

Run this scenario in the playground

One batch, one request (zstd). t+3.600s

linger.ms flushes the whole 6-record batch at once - offsets 6..11, a single produce request. Same data as Half 1, but fewer and larger batches: the throughput win compression buys.

Open at this step: step 6, One batch, one request (zstd)

What you start with

compression.type is the throughput knob: the producer compresses each batch, so more records fit inside the same batch.size byte budget before the size trigger fires. The same six-record burst runs twice - first with compression off (two 3-record batches) and then with zstd (one larger batch on the linger trigger).

Step by step

One stream, two settings t+0.000s p1 has batch.size=3 and compression.type=none. Watch how the SAME six-record burst is shaped first without compression, then with zstd. Open at this step: step 1, One stream, two settings
Batch 1 of 2 (no compression) t+1.100s A1-A3 fill batch.size=3, so the batch size-triggers a flush immediately - offsets 0,1,2. With compression OFF, 3 records is a full batch. Open at this step: step 2, Batch 1 of 2 (no compression)
Batch 2 of 2 (no compression) t+1.300s A4-A6 fill a second 3-record batch - offsets 3,4,5. Six records → TWO produce requests. Here batch.size=3 caps every batch at 3 records (the simulator counts records; real batch.size is a byte budget). Open at this step: step 3, Batch 2 of 2 (no compression)
Switch on zstd t+2.500s A config_change sets compression.type=zstd (live in the simulator; a real producer would be recreated) with ratio ≈ 0.3. The effective batch.size becomes ⌊3 / 0.3⌋ = 10 records - the same byte budget now holds far more records. Open at this step: step 4, Switch on zstd
Compression packs the batch t+3.300s The identical six-record burst (B1-B6) accumulates in ONE batch - past the old batch.size of 3 - because the 10-record effective cap is never hit. The fill bar climbs without flushing. Open at this step: step 5, Compression packs the batch
One batch, one request (zstd) t+3.600s linger.ms flushes the whole 6-record batch at once - offsets 6..11, a single produce request. Same data as Half 1, but fewer and larger batches: the throughput win compression buys. Open at this step: step 6, One batch, one request (zstd)

The takeaway

Compression buys throughput - fewer, larger batches and fewer produce requests for the same data, at a little CPU cost - while changing batch shape only: every record still lands exactly once, in produce order.

Configs in this scenario

compression.type The codec the producer compresses each batch with: none (the default), gzip, snappy, lz4 or zstd; the batch stays compressed on the broker and is decompressed by the consumer. How compression.type works
batch.size The most bytes the producer puts in one batch for a partition (default 16384); a batch goes out when it is full or when linger.ms runs out, whichever comes first. How batch.size works
linger.ms How long the producer waits for more records before it sends a batch that is not full (default 5 ms since Kafka 4.0, 0 before). How linger.ms works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.2.3" title="Kafka Simulator: Compression and batch shape" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/compression-and-batch-shape/">Kafka compression.type: fewer, larger producer batches</a></p>

More in Producers & EOS

All scenarios