Scenario 01.2.4 · Backpressure: buffer.memory and max.block.ms

Kafka buffer.memory and max.block.ms: producer backpressure

Producers & EOS · Intermediate · about 3 min · Apache Kafka 4.3

buffer.memory caps the producer's un-acked send buffer: when slow acks fill it, the next send BLOCKS for up to max.block.ms - and if the buffer never drains, the send times out.

What you start with

buffer.memory is the producer's send-buffer budget for records waiting to be sent or still awaiting their ack; completed batches free space. p1 is capped at 2048 bytes (room for 2 un-acked records) writing acks=all behind two throttled followers, so the high watermark never advances and no ack ever returns. Two records fill the buffer, the third send blocks (backpressure) for max.block.ms, then times out when the buffer never drains.

Step by step

1. Cluster ready t+0.000s p1 writes acks=all with buffer.memory = 2048 B (room for 2 un-acked records) and max.block.ms = 600. The buffer starts empty. Open at this step: step 1, Cluster ready
2. Followers throttled t+0.300s Both followers (broker-2, broker-3) are slow, so the high watermark will never advance - meaning acks=all can never commit and no ack will return to free the buffer. Open at this step: step 2, Followers throttled
3. A1 buffered, not acked t+0.600s A1 lands on the leader's log (offset 0) but the slow followers never fetch, so HW stays 0 and A1 is never acked. The un-acked buffer holds 1024 B; p1 sits in waiting. Open at this step: step 3, A1 buffered, not acked
4. Buffer full (2 records) t+0.800s A2 lands on the leader (offset 1), still un-acked. The buffer now holds 2048 B = exactly buffer.memory. The next send has no room. Open at this step: step 4, Buffer full (2 records)
5. Send blocked (backpressure) t+1.200s A3's send finds the buffer full (2048 >= 2048), so it BLOCKS - p1 goes to blocked, holding A3 while max.block.ms counts down (fires at 1600 ms). A3 is not appended; the log still holds only [0, 1]. Open at this step: step 5, Send blocked (backpressure)
6. Still blocked, no ack t+1.500s Mid-block: no ack has arrived (the followers are still slow), so the buffer has not drained. The send is still blocked, the timer still counting down. Open at this step: step 6, Still blocked, no ack
7. Block timed out t+1.700s max.block.ms elapsed with the buffer still full, so A3's send FAILS with a BufferExhaustedException (a TimeoutException; the simulator shows it as request_timed_out). p1 goes to error and the held record is dropped - backpressure surfaced the stall instead of buffering without limit. Open at this step: step 7, Block timed out

The takeaway

Backpressure is the producer protecting itself: when the broker can't keep up, a full buffer.memory makes the next send block for up to max.block.ms instead of buffering without limit - and past that window the stalled send fails with a TimeoutException rather than growing memory unbounded.

Configs in this scenario

• buffer.memory The bytes the producer may hold for records waiting to be sent or still awaiting an acknowledgement (default 33554432, 32 MiB); when it is full, send() blocks for up to max.block.ms and then fails with BufferExhaustedException.
• max.block.ms How long send(), partitionsFor() and the transaction methods may block, waiting for metadata, for room in buffer.memory or for the transaction coordinator, before they throw a TimeoutException (default 60000, 1 minute).
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=01.2.4" title="Kafka Simulator: Backpressure: buffer.memory and max.block.ms" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/backpressure-buffer-memory/">Kafka buffer.memory and max.block.ms: producer backpressure</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• TimeoutException in Kafka producers: causes and fixes

More in Producers & EOS

• Kafka producer acks=0: fire and forget, silent data loss
• Kafka producer acks=1: acknowledged by the leader only
• Kafka acks=0 vs acks=1 vs acks=all on a degraded ISR
• Kafka acks=1 data loss: min.insync.replicas does not help
• Kafka linger.ms vs batch.size: when the producer sends
• Kafka producer partitioners: keyed, round-robin and sticky
• Kafka compression.type: fewer, larger producer batches
• Kafka idempotent producer: retrying NOT_ENOUGH_REPLICAS
• Kafka enable.idempotence=false: a retry writes a duplicate
• Kafka idempotent producer dedup vs a plain producer
• Kafka max.in.flight.requests.per.connection and ordering

All scenarios