Scenario 04.3.1 · Consume-process-produce pipeline

Kafka consume-process-produce loop with exactly-once

Transactions · Advanced · about 3 min · Apache Kafka 4.3

The EOS read-process-write loop: poll the input, produce the output inside a transaction, stage the input offsets with send_offsets_to_txn, and commit - output and offsets land atomically.

What you start with

The exactly-once read-process-write loop that Kafka Streams runs under the hood. The pipeline polls an input topic, produces the processed result to an output topic inside a transaction, and stages the consumed input offsets in that same transaction with send_offsets_to_txn before committing.

Step by step

1. Cluster ready · pipeline t+0.000s Two topics - `orders-in` (input) and `orders-out` (output) - a read_committed consumer group on the input, and a transactional producer for the output. The classic EOS read-process-write topology. Open at this step: step 1, Cluster ready · pipeline
2. Input seeded · orders-in t+0.600s A plain producer writes two records to `orders-in`. Non-transactional writes are immediately stable, so the read_committed pipeline consumer can poll them. Open at this step: step 2, Input seeded · orders-in
3. Polled · processed t+1.500s The `pipeline` group polled `orders-in` and advanced its position to 2 - the two input records have been "processed". This read position is exactly what must be committed. Open at this step: step 3, Polled · processed
4. begin output txn t+1.650s The output producer opens a transaction. From here until commit, the output write AND the input-offset commit are one atomic unit. Open at this step: step 4, begin output txn
5. Output produced · uncommitted t+2.300s The processed result is written to `orders-out` (HW = 1) but held below the LSO (0): durable in the log, not yet visible to a read_committed reader downstream. Open at this step: step 5, Output produced · uncommitted
6. Offsets staged t+2.500s send_offsets_to_txn stages the consumer position {0:2} INSIDE the transaction (txnOffsets). The offset commit is now part of the txn - not a separate write that could succeed or fail on its own. Open at this step: step 6, Offsets staged
7. commit · output + offsets t+3.400s commit_transaction makes both durable atomically: the output LSO advances to 1 (visible) AND the input offsets commit to {0:2}. A crash can never leave output-without-offsets (a duplicate) or offsets-without-output (a loss). Open at this step: step 7, commit · output + offsets

The takeaway

Because the output records and the input offsets commit atomically, a crash can never leave output written with input un-committed (a duplicate) or input committed with output missing (a loss) - the whole consume→process→produce→commit cycle is one atomic unit.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=04.3.1" title="Kafka Simulator: Consume-process-produce pipeline" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/consume-process-produce/">Kafka consume-process-produce loop with exactly-once</a></p>

Further reading

Apache Kafka documentation

More in Transactions

• Kafka at-most-once vs at-least-once vs exactly-once
• At-least-once delivery in Kafka: where duplicates come from
• Kafka transaction commit vs abort with read_committed
• Kafka isolation.level: read_committed vs read_uncommitted
• Kafka last stable offset (LSO) with concurrent transactions
• Kafka transaction.timeout.ms: the coordinator aborts
• Kafka read_committed during a leader election
• Kafka zombie producer fencing with transactional.id
• Kafka processing.guarantee=exactly_once_v2 end to end

All scenarios