Scenario 13.1.3 · A partition on disk, leader vs follower
Kafka log segments on disk: leader vs follower
A partition is a directory of segment files named by base offset: watch them roll, see a stalled follower hold fewer bytes than its leader, and watch retention delete a whole file.
Run this scenario in the playgroundAn empty log directory. t+0.000s
Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). You start on broker-1, inside its log root. `tree` draws what a broker actually keeps on disk: one directory per partition replica it hosts, and inside it an active segment file that already exists while holding nothing.
Open at this step: step 1, An empty log directorySix records, and the log has rolled. t+2.700s
Six records are in. `ls -lh orders-0` shows two segment files where there was one: a file is named after the FIRST offset it can hold, zero padded to twenty digits, so the names are a map of the log. Open the closed one with `kafka-dump-log --files orders-0/00000000000000000000.log --print-data-log` and read the batch header and the records inside it.
Open at this step: step 2, Six records, and the log has rolledNine records, three segments. t+4.200s
Three more records rolled a third file. `ls -lh orders-0` lists them with the sizes they carry, `kafka-dump-log --files orders-0/00000000000000000004.log --print-data-log` reads the middle one, and `du -sh orders-0` totals what this partition costs broker-1.
Open at this step: step 3, Nine records, three segmentsThe same partition, on a stalled follower. t+4.600s
broker-3 stopped fetching in the middle of the stream. `ssh broker-3` stands you on the follower, then run `ls -lh orders-0` there: same directory, same naming, less data. Its newest segment is short and the leader's third file does not exist on this machine at all, and `du -sh orders-0` prices the difference. A replica this far behind is also out of the ISR, which `kafka-topics --describe --topic orders` confirms; at RF 3 with min.insync.replicas 2 the writes kept succeeding anyway.
Open at this step: step 4, The same partition, on a stalled followerRetention deleted a file, not records. t+8.200s
A retention run, scripted here rather than triggered by the 60 second retention.ms window, advanced the log start to offset 4, which is a segment boundary. `exit` back to broker-1 and run `ls orders-0`: the first segment did not shrink, it is gone, and so are its index files. `kafka-get-offsets --topic orders --time earliest` reports the new floor the deletion left behind.
Open at this step: step 5, Retention deleted a file, not recordsWhat you start with
This scenario is played from the terminal (Ctrl and the backtick key). A partition is not an abstraction on disk: it is a directory of segment files, one per broker that hosts a replica, and the file names are offsets. Nine records roll three segments, a stalled follower falls behind mid-stream, and then retention runs.
Step by step
- An empty log directory t+0.000s Open the terminal with Ctrl and the backtick key (on a phone, the "..." menu). You start on broker-1, inside its log root. `tree` draws what a broker actually keeps on disk: one directory per partition replica it hosts, and inside it an active segment file that already exists while holding nothing. Open at this step: step 1, An empty log directory
- Six records, and the log has rolled t+2.700s Six records are in. `ls -lh orders-0` shows two segment files where there was one: a file is named after the FIRST offset it can hold, zero padded to twenty digits, so the names are a map of the log. Open the closed one with `kafka-dump-log --files orders-0/00000000000000000000.log --print-data-log` and read the batch header and the records inside it. Open at this step: step 2, Six records, and the log has rolled
- Nine records, three segments t+4.200s Three more records rolled a third file. `ls -lh orders-0` lists them with the sizes they carry, `kafka-dump-log --files orders-0/00000000000000000004.log --print-data-log` reads the middle one, and `du -sh orders-0` totals what this partition costs broker-1. Open at this step: step 3, Nine records, three segments
- The same partition, on a stalled follower t+4.600s broker-3 stopped fetching in the middle of the stream. `ssh broker-3` stands you on the follower, then run `ls -lh orders-0` there: same directory, same naming, less data. Its newest segment is short and the leader's third file does not exist on this machine at all, and `du -sh orders-0` prices the difference. A replica this far behind is also out of the ISR, which `kafka-topics --describe --topic orders` confirms; at RF 3 with min.insync.replicas 2 the writes kept succeeding anyway. Open at this step: step 4, The same partition, on a stalled follower
- Retention deleted a file, not records t+8.200s A retention run, scripted here rather than triggered by the 60 second retention.ms window, advanced the log start to offset 4, which is a segment boundary. `exit` back to broker-1 and run `ls orders-0`: the first segment did not shrink, it is gone, and so are its index files. `kafka-get-offsets --topic orders --time earliest` reports the new floor the deletion left behind. Open at this step: step 5, Retention deleted a file, not records
The takeaway
A segment file is named after the FIRST offset it can hold, so the file names are a map of the log. The directory belongs to a REPLICA, not to the partition: a follower that stops fetching keeps a shorter active segment than its leader, and the only way to see that is to stand on the follower. And retention works in whole files - advancing the log start to a segment boundary makes a file disappear from ls rather than shrink, which is why a retention window can only ever be approximate.
Configs in this scenario
-
min.The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas worksinsync. replicas -
retention.How long a topic with cleanup.policy=delete keeps a log segment before it may be deleted (default 604800000, 7 days); -1 removes the time limit. How retention.ms worksms
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=13.1.3" title="Kafka Simulator: A partition on disk, leader vs follower" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/partition-on-disk/">Kafka log segments on disk: leader vs follower</a></p>Further reading
More in Terminal ops
- Kafka from the terminal: hosts, ssh and the KRaft controller
- Diagnose consumer lag with kafka-consumer-groups --describe
- Under-replicated vs under-min-ISR partitions in Kafka
- Reset Kafka consumer group offsets with --to-earliest
- Set retention.ms on a live topic with kafka-configs
- Kafka partition reassignment: execute, throttle, verify
- Graceful broker shutdown with kafka-server-stop
- Kafka tools on two clusters: --bootstrap-server decides