---
title: "Kafka consume-process-produce loop with exactly-once"
description: "Poll orders-in, write to orders-out inside a transaction and stage input offsets with sendOffsetsToTransaction. Kafka commits output and offsets atomically."
url: https://monedula.dev/kafka-simulator/scenarios/consume-process-produce/
---

# Kafka consume-process-produce loop with exactly-once

Scenario 04.3.1 · Consume-process-produce pipeline. Transactions · Advanced · about 3 min · Apache Kafka 4.3.

The EOS read-process-write loop: poll the input, produce the output inside a transaction, stage the input offsets with send_offsets_to_txn, and commit - output and offsets land atomically.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1

## What you start with

The exactly-once read-process-write loop that Kafka Streams runs under the hood. The pipeline polls an input topic, produces the processed result to an output topic inside a transaction, and stages the consumed input offsets in that same transaction with send_offsets_to_txn before committing.

## Step by step

1. **Cluster ready · pipeline** (t+0.000s): Two topics - `orders-in` (input) and `orders-out` (output) - a read_committed consumer group on the input, and a transactional producer for the output. The classic EOS read-process-write topology. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=0)
2. **Input seeded · orders-in** (t+0.600s): A plain producer writes two records to `orders-in`. Non-transactional writes are immediately stable, so the read_committed pipeline consumer can poll them. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=1)
3. **Polled · processed** (t+1.500s): The `pipeline` group polled `orders-in` and advanced its position to 2 - the two input records have been "processed". This read position is exactly what must be committed. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=2)
4. **begin output txn** (t+1.650s): The output producer opens a transaction. From here until commit, the output write AND the input-offset commit are one atomic unit. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=3)
5. **Output produced · uncommitted** (t+2.300s): The processed result is written to `orders-out` (HW = 1) but held below the LSO (0): durable in the log, not yet visible to a read_committed reader downstream. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=4)
6. **Offsets staged** (t+2.500s): send_offsets_to_txn stages the consumer position {0:2} INSIDE the transaction (txnOffsets). The offset commit is now part of the txn - not a separate write that could succeed or fail on its own. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=5)
7. **commit · output + offsets** (t+3.400s): commit_transaction makes both durable atomically: the output LSO advances to 1 (visible) AND the input offsets commit to {0:2}. A crash can never leave output-without-offsets (a duplicate) or offsets-without-output (a loss). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=04.3.1&t=6)

## The takeaway

Because the output records and the input offsets commit atomically, a crash can never leave output written with input un-committed (a duplicate) or input committed with output missing (a loss) - the whole consume→process→produce→commit cycle is one atomic unit.

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#message-delivery-semantics)