Kafka troubleshooting

TimeoutException (REQUEST_TIMED_OUT)

Kafka error · REQUEST_TIMED_OUT · retriable

org.apache.kafka.common.errors.TimeoutException

The producer gave up on a record because a time limit ran out: send() could not get topic metadata or buffer space within max.block.ms, or the record was not acknowledged within delivery.timeout.ms ("Expiring N record(s)"). The message says which limit fired; fix what made the brokers slow or unreachable instead of only raising the timeout.

Reproduce it in the playground

Scenario 01.2.4 · Backpressure: buffer.memory and max.block.ms

The moment it happensBlock timed out. t+1.700s

max.block.ms elapsed with the buffer still full, so A3's send FAILS with a BufferExhaustedException (a TimeoutException; the simulator shows it as request_timed_out). p1 goes to error and the held record is dropped - backpressure surfaced the stall instead of buffering without limit.

Open at this step: step 7, Block timed out

What it means

org.apache.kafka.common.errors.TimeoutException covers several timers, and the message tells them apart. "Topic orders not present in metadata after 60000 ms." means send() waited max.block.ms (default 60 seconds) for metadata. "Failed to allocate 1024 bytes within the configured max blocking time 60000 ms" comes from BufferExhaustedException, a subclass: buffer.memory (33554432 bytes by default) was full of records waiting to be sent or acknowledged. In both cases send() blocks the calling thread for up to max.block.ms, then returns a failed Future and calls the callback. "Expiring 1 record(s) for orders-0:120001 ms has passed since batch creation" means the batch was not acknowledged within delivery.timeout.ms (default 2 minutes), the budget for lingering, sending, retries and the ack. Inside that budget, a request with no response after request.timeout.ms (default 30 seconds) is failed as REQUEST_TIMED_OUT ("Disconnected from node 1 due to timeout") and retried, and the broker itself answers REQUEST_TIMED_OUT when an acks=all write waited longer than that timeout for the in-sync replicas. Only the request timeout carries the protocol error REQUEST_TIMED_OUT (code 7); the max.block.ms and delivery.timeout.ms cases are raised by the client and have no broker error code. Scenario 01.2.4 shows the buffer case: the third send blocks, then fails after max.block.ms.

Likely causes, most common first

The topic or partition is not in the metadata. The topic does not exist and auto.create.topics.enable is off on the brokers, the name is misspelled, a record names a partition number the topic does not have ("Partition 7 of topic orders with partition count 6 is not present in metadata after 60000 ms."), or no bootstrap server answers.
Records are produced faster than they are delivered. Slow or distant brokers, client quotas that throttle the producer, or an acks=all path waiting on slow followers let unacknowledged records pile up in buffer.memory. Once it is full, every send() blocks up to max.block.ms and then fails.
The partition leader cannot be reached or does not answer. The client gets metadata from a bootstrap server but cannot reach the leader: advertised.listeners resolve to an address the client cannot route to, a firewall drops the connection, or the leader is overloaded or paused. Requests time out, are retried, and the batch finally expires.
acks=all waits on followers that are in sync but slow. The leader holds the response until every in-sync replica has the batch. If that takes longer than request.timeout.ms, the request times out with REQUEST_TIMED_OUT, from the broker or from the client, and repeated timeouts end in an expired batch. When the ISR is too small instead, the error is NOT_ENOUGH_REPLICAS.

How to fix it

Read the message first: "not present in metadata" points at the topic name, its existence or bootstrap.servers; "Failed to allocate" at throughput and buffer.memory; "Expiring N record(s)" at the path to the partition leader.
Check the producer metrics: buffer-available-bytes and bufferpool-wait-ratio show a full buffer, record-retry-rate and record-error-rate show retries and failures, request-latency-max shows slow brokers.
Test connectivity from the client host to every broker's advertised listener, not only to the bootstrap address.
Keep delivery.timeout.ms at least request.timeout.ms plus linger.ms, as the producer docs ask, and long enough to ride out a leader election or a broker restart.
Raise buffer.memory only for short bursts that the brokers absorb on average; for a lasting imbalance add partitions, brokers or producer instances, or slow the source down.
Decide what a failed send means for the application (log and drop, resend, or stop) and handle the exception in every callback.

What not to do

Setting max.block.ms or delivery.timeout.ms very high to make the exception go away: send() then blocks the calling thread for longer and failures surface later, while the cause stays.

Does the client retry it?

The producer already retried or waited. Request timeouts and the broker's REQUEST_TIMED_OUT are retried until delivery.timeout.ms runs out, and a max.block.ms timeout means send() already waited that long for metadata or buffer space. When the exception reaches your callback, those records have failed; sending them again is the application's choice and writes a duplicate if the first attempt did reach the broker.

Configs involved

max.block.ms How long send(), partitionsFor() and the transaction methods may block, waiting for metadata, for room in buffer.memory or for the transaction coordinator, before they throw a TimeoutException (default 60000, 1 minute).
buffer.memory The bytes the producer may hold for records waiting to be sent or still awaiting an acknowledgement (default 33554432, 32 MiB); when it is full, send() blocks for up to max.block.ms and then fails with BufferExhaustedException.
delivery.timeout.ms The upper bound on reporting success or failure after send() returns, covering the wait in a batch, the wait for the acknowledgement and every retry (default 120000, 2 minutes); it should be at least request.timeout.ms plus linger.ms.
request.timeout.ms How long the client waits for the response to one request before it sends it again or fails it (default 30000, 30 seconds); for a producer it should stay above the broker replica.lag.time.max.ms to avoid retries that write duplicates.
linger.ms How long the producer waits for more records before it sends a batch that is not full (default 5 ms since Kafka 4.0, 0 before).
acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence).
retries How many times the producer sends a batch again after a transient error (default 2147483647, so in practice delivery.timeout.ms ends the retrying); idempotence needs it above 0.

Reproduce it in the simulator

Kafka buffer.memory and max.block.ms: producer backpressure (Backpressure: buffer.memory and max.block.ms) Open at the moment it happens: Kafka buffer.memory and max.block.ms: producer backpressure

Sources

NotEnoughReplicasException in Kafka: causes and fixes

All troubleshooting pages