---
title: "Kafka log truncation when an old leader returns"
description: "Two uncommitted records sit only on the leader when it is cut off. A new Kafka leader takes over at HW 2, and the old one truncates them before rejoining."
url: https://monedula.dev/kafka-simulator/scenarios/returning-replica-truncation/
---

# Kafka log truncation when an old leader returns

Scenario 03.2.5 · Returning replica and log truncation. Replication · Advanced · about 4 min · Apache Kafka 4.3.

A replica holding records the new leader never got truncates them at the divergence point (found via leader epochs) before it can rejoin the ISR - the lossless guard against divergent logs.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5

## What you start with

A replica may hold records BEYOND the high watermark - uncommitted writes the leader appended but the ISR never fetched. Here both followers are throttled so O3,O4 land only on the leader; the leader is then isolated and a follower at the HW is elected, orphaning that uncommitted tail.

## Step by step

1. **Three replicas in sync** (t+0.000s): RF=3, min.insync.replicas=2. broker-1 leads; broker-2 and broker-3 are followers. All three are in the ISR at LEO 0 - every replica holds an identical, empty log. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=0)
2. **O1, O2 committed (HW=2)** (t+5.000s): Two records replicate to all three replicas, so the high watermark advances to 2 and the acks=all producer is acked. HW=2 is the durability frontier: only offsets below it are committed and readable. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=1)
3. **O3 lands on the leader only** (t+9.000s): broker-2 and broker-3 are throttled (frozen at LEO 2), so O3 reaches only the leader: broker-1 LEO climbs to 3 while the HW stays pinned at 2. The record is uncommitted - and acks=all leaves the producer waiting, never telling it the write is durable. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=2)
4. **Leader holds an uncommitted tail** (t+11.000s): O4 also lands on broker-1 alone (LEO 4). The leader now carries two records - offsets 2 and 3 - that no other in-sync replica has and that were never committed. This is the divergence hazard: a replica ahead of the high watermark. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=3)
5. **Partition isolates broker-1** (t+14.000s): A network partition isolates broker-1. Once it loses its controller session, the controller fences it and elects broker-2, the first remaining ISR replica (LEO 2, leader epoch bumps to 1), and broker-1 drops out of the ISR - taking its uncommitted offsets 2,3 with it. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=4)
6. **Return → log truncation** (t+18.000s): The network heals. broker-1 returns with LEO 4, ahead of the leader's LEO 2 - a divergent log. Using leader epochs it learns epoch 0 ended at offset 2 on the new leader and TRUNCATES there (offsets 2,3 discarded: the log_truncate moment), then refetches from the leader. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=5)
7. **Rejoined, logs consistent** (t+22.000s): broker-1 has caught up to LEO 2 and is back in the ISR. Every replica holds the same committed log again. No committed record was ever lost - truncation only dropped writes that never crossed the high watermark, which is exactly why they were safe to drop. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=03.2.5&t=6)

## The takeaway

When the old leader returns it truncates its log where it diverges from the new leader (found via leader epochs; here offset 2, the HW) before rejoining the ISR - only never-committed records are dropped, so the cluster reconverges without losing a single acked write.

## Configs in this scenario

- [`min.insync.replicas`](https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_min.insync.replicas): The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. [How min.insync.replicas works](https://monedula.dev/kafka-simulator/configs/min-insync-replicas/)
- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#replication)