Kafka producer config
batch.size: the producer batch limit
batch.size is the most bytes the Kafka producer puts in one batch for a single partition (default 16384, 16 KiB). A batch is sent when it is full or when linger.ms runs out, whichever comes first, so batch.size caps a batch rather than making the producer wait for one.
- Config
batch.(producer)size - Default
- 16384 (bytes)
- Valid values
- 0 or more; 0 turns batching off
What it does
Each partition the producer writes to has its own open batch, carved out of buffer.memory. Records are appended until the next one would not fit, and then the batch is ready to send. Ready batches for partitions led by the same broker travel together in one produce request. A single record larger than batch.size gets a batch of its own. With compression on, the producer counts the estimated compressed size against batch.size, so a compressed batch holds more records.
The trade-off
A larger batch.size means fewer requests and better compression, so more throughput and less broker load, but every partition in use can hold a batch of that size, so many partitions times a large batch.size can exhaust buffer.memory. A smaller one sends more, smaller requests. On its own a larger batch.size adds no latency: only linger.ms makes the producer wait.
Advice
- For throughput, raise batch.size (64 KiB to 256 KiB is common) together with linger.ms; one without the other changes little.
- Check buffer.memory against the number of partitions the producer writes to at once.
- Do not set 0: every record then goes out in a batch of its own.
See it in the simulator
- Kafka linger.ms vs batch.size: when the producer sends (linger.ms and batch.size) Step 4: batch.size reached · 3-record batch flushed at once Open at this step: Kafka linger.ms vs batch.size: when the producer sends
- Kafka compression.type: fewer, larger producer batches (Compression and batch shape) Step 3: Batch 2 of 2 (no compression) Open at this step: Kafka compression.type: fewer, larger producer batches