Kafka troubleshooting
ProducerFencedException (PRODUCER_FENCED)
org.
Another producer instance called initTransactions() with the same transactional.id, so the transaction coordinator bumped the producer epoch and this older instance may no longer write or commit. Close the fenced producer, and give each live instance its own stable transactional.id. On the original transaction protocol (transaction.version=1), a transaction that timed out also ends in PRODUCER_FENCED, with no second instance involved.
Reproduce it in the playground3 steps beforetxn 1 committed · read, no auto-commit. t+1.900s
commit advances the LSO and eos reads both records (position 2). But committed stays at 0: enable.auto.commit is off, so offsets move only through the transaction.
Open at this step: step 4, txn 1 committed · read, no auto-commit2 steps beforetxn 2 · offsets staged. t+2.900s
In a second txn tx produced an output record and called sendOffsetsToTransaction(eos): eos offsets {0:2} are staged inside the transaction, not yet committed.
Open at this step: step 5, txn 2 · offsets staged1 step beforecommit · offsets + output atomic. t+3.400s
commit_transaction commits the output record AND eos input offsets atomically - eos.committed jumps to {0:2}. On an abort neither would commit: exactly-once.
Open at this step: step 6, commit · offsets + output atomicThe moment it happenszombie fenced. t+3.900s
tx-zombie (epoch 0) tries to open a new transaction, but tx holds a newer epoch, so the zombie is FENCED and can never write duplicates. In Kafka the fence lands on its first send or commit, since beginTransaction() is client-side, and transaction.version=2 bumps the epoch at every commit.
Open at this step: step 7, zombie fencedWhat it means
The transaction coordinator keeps one producer id and an epoch per transactional.id. initTransactions() completes or aborts whatever the previous holder left open and takes a new epoch, and from then on only that epoch may act. A zombie, an instance that was paused or cut off while a replacement started, is turned away the next time it talks to the coordinator: commitTransaction(), abortTransaction() and sendOffsetsToTransaction() fail with PRODUCER_FENCED (error code 90), "There is a newer producer with the same transactionalId which fences the current one." Every later call on that producer rethrows it as "Producer with transactionalId 'orders-tx' and (producerId=1000, epoch=3) has been fenced by another producer with the same transactionalId". Partition leaders cannot tell why an epoch moved, so a zombie's send() gets INVALID_PRODUCER_EPOCH (error code 47) and InvalidProducerEpochException ("Producer attempted to produce with an old epoch.") instead, a split KIP-588 introduced in Kafka 2.7. In Kafka 4.x both exceptions extend ApplicationRecoverableException: the producer cannot be used again, and the application has to recreate it and rewind to its last committed state. Fencing is working as designed here; it is what stops the old instance from writing duplicates into the stream.
Likely causes, most common first
- Two live instances share a transactional.id. Old and new versions overlap during a deployment, a scaled-out service copies one static transactional.id into every replica, or a second application points at the same cluster with the same id. Each initTransactions() fences the other, so they take turns failing.
- A zombie came back after its replacement started. A long GC pause, a network partition or a frozen container outlasted whatever detected the failure and started a replacement. The replacement initialized the same transactional.id, and the old instance is fenced as soon as it resumes. Scenarios 04.3.2 and 04.3.3 show this case; the simulator fences the zombie when it calls beginTransaction(). A real beginTransaction() sends nothing, so a real zombie is fenced at its first send or commit.
- A transaction timed out on the original protocol. When a transaction runs past transaction.timeout.ms, the coordinator aborts it and bumps the epoch. On the original transaction protocol (transaction.version=1), a later commitTransaction() or abortTransaction() from that producer gets PRODUCER_FENCED although no other instance exists, the case KIP-588 describes.
How to fix it
- On ProducerFencedException, close() the producer and stop the work it was doing; abortTransaction() fails the same way, and the open transaction was already aborted by whatever fenced this instance.
- Give every concurrently running producer its own transactional.id that stays the same across its restarts. The KafkaProducer Javadoc suggests deriving it from the shard identifier of a partitioned application, so exactly one instance owns each id.
- Make deployments hand over cleanly: stop the old instance before the new one calls initTransactions(), or treat the fencing of the old one as the expected way it shuts down.
- Inspect the id with kafka-transactions.sh --bootstrap-server localhost:9092 describe --transactional-id my-txn-id, which prints the ProducerId, ProducerEpoch, TransactionState and how long the current transaction has been open.
- If fencing follows long pauses, find the pause (GC logs, a stalled poll loop) and keep each transaction well inside transaction.timeout.ms.
What not to do
- Restarting the fenced instance automatically with the same transactional.id while the other one is still alive: each start fences the other, and the two keep knocking each other out.
- Generating a random transactional.id on every start to avoid the error: a restarted instance then no longer fences its own zombie, which removes the protection against duplicates.
Does the client retry it?
No. A fenced producer stays fenced: every later call fails the same way, so the only step left is close(). A new producer with the same transactional.id can take over, but its initTransactions() fences whichever instance held the id before.
Configs involved
transactional.A stable name that turns on transactions for a producer (and implies idempotence); a new producer that calls initTransactions() with the same id fences the older one.id transaction.The longest a transaction may stay open, counted from the first partition added to it, before the transaction coordinator aborts it (default 60000, 1 minute); it may not exceed the broker transaction.max.timeout.ms.timeout. ms enable.Gives the producer an id and a sequence number per partition so the broker drops duplicate retries and keeps them in order; on by default unless a conflicting setting (acks other than all, retries=0, more than 5 in-flight requests) turns it off.idempotence isolation.read_uncommitted (the default) returns every record, including those of open and aborted transactions; read_committed returns only committed transactional records and stops at the last stable offset.level
Reproduce it in the simulator
- Kafka zombie producer fencing with transactional.id (EOS - offsets in a txn + fencing) Open at the moment it happens: Kafka zombie producer fencing with transactional.id
- Kafka processing.guarantee=exactly_once_v2 end to end (exactly_once_v2 end-to-end) Open at the moment it happens: Kafka processing.guarantee=exactly_once_v2 end to end
Sources
- Apache Kafka 4.3 Javadoc: ProducerFencedException
- Apache Kafka 4.3 Javadoc: InvalidProducerEpochException
- Apache Kafka 4.3.0 source: Errors.java (PRODUCER_FENCED 90, INVALID_PRODUCER_EPOCH 47)
- KIP-588: Allow producers to recover gracefully from transaction timeouts
- Apache Kafka 4.3 Javadoc: KafkaProducer (initTransactions, transactional.id)
- Apache Kafka 4.3: producer config transactional.id
- Apache Kafka 4.3: using transactions (exception categories)
- Apache Kafka 4.3.0 source: TransactionManager.java (beginTransaction() is a local state change; fenced and old-epoch messages on later calls)
- Apache Kafka 4.3.0 source: TransactionCoordinator.scala (epoch checks on EndTxn)
- Apache Kafka 4.3.0 source: ProducerAppendInfo.java (INVALID_PRODUCER_EPOCH on produce since 2.7)
- Apache Kafka 4.3.0 source: TransactionsCommand.java (kafka-transactions.sh describe)