Kafka producer config

compression.type: gzip vs snappy vs lz4 vs zstd

Producer config · default none (producer); producer (topic) · Apache Kafka 4.3

compression.type sets the codec the Kafka producer compresses each batch with: none (the default), gzip, snappy, lz4 or zstd. Compression works per batch, so it pays off most when linger.ms and batch.size let batches grow. The topic has its own compression.type, which by default keeps whatever the producer sent.

Config: compression.type (producer)
Default: none (producer); producer (topic)
Valid values: Producer: none, gzip, snappy, lz4, zstd. Topic: producer, uncompressed, gzip, snappy, lz4, zstd
Also set at: The topic config compression.type (broker-wide default: compression.type) decides how the broker stores batches: producer keeps the producer codec, and any other value makes the broker recompress batches that arrive in a different codec.
Reference: Apache Kafka documentation for compression.type

Scenario 01.2.3 · Compression and batch shape step 6

One batch, one request (zstd). linger.ms flushes the whole 6-record batch at once - offsets 6..11, a single produce request. Same data as Half 1, but fewer and larger batches: the throughput win compression buys. Open this step in the full playground

Values

none No compression: the cheapest on CPU, the most bytes on the network and on disk.
gzip A high ratio at the highest CPU cost of the four; its level is set with compression.gzip.level.
snappy Fast with a modest ratio, and no level setting.
lz4 Very fast with a ratio similar to snappy; its level is set with compression.lz4.level.
zstd Usually the best ratio for the CPU it spends; its level is set with compression.zstd.level.

What it does

The producer compresses each record batch as a whole before sending it, and the batch stays compressed on the broker disk, in replication and on the way to consumers, which decompress it. The ratio depends on how much similar data is in one batch, so tiny batches compress poorly whatever the codec. Because the producer counts the estimated compressed size against batch.size, compression also lets more records into each batch.

The trade-off

Compression spends producer and consumer CPU to save network bandwidth, broker disk and page cache. gzip saves the most bytes for the most CPU, snappy and lz4 the least CPU for fewer bytes saved, and zstd sits in between with a strong ratio. A topic compression.type that is neither producer nor the producer codec makes the broker recompress every batch, which costs broker CPU.

Advice

Start with zstd or lz4 and measure on your own data; JSON and text compress far better than payloads that are already compressed, such as images.
Leave the topic compression.type at producer so the broker stores batches as they arrive.
Raise linger.ms and batch.size along with compression; a batch of one record barely compresses.

See it in the simulator

Kafka compression.type: fewer, larger producer batches (Compression and batch shape) Step 6: One batch, one request (zstd) Open at this step: Kafka compression.type: fewer, larger producer batches
Tune Kafka throughput: linger.ms, batch.size and zstd (Capstone: tune a high-throughput, low-latency pipeline) Step 3: One compressed batch flushed - HW climbs Open at this step: Tune Kafka throughput: linger.ms, batch.size and zstd

Related configs

All Kafka configs