How it works

Kafka’s threads only record metric references in memory; the reporter’s own threads do all export I/O.

Kafka MetricsReporter SPI  ->  MetricRegistry        --+
Kafka Yammer registry      ->  YammerMetricRegistry  --+->  export thread  ->  OTLP exporter  ->  collector
Kafka MetricsContext       ->  resource attributes   --+

Where it plugs in

Kafka loads the reporter through metric.reporters, as an implementation of its MetricsReporter interface. Kafka then calls it at a few points:

• init and metricChange when metrics are registered or updated, and metricRemoval when they go away.
• contextChange with the process’s metrics context: on a broker, the cluster ID and the node ID (or broker ID in ZooKeeper mode), and the namespace that becomes the prefix of metric names.
• configure and close for its settings and lifecycle.

Kafka keeps several broker-internal metrics, such as UnderReplicatedPartitions, OfflinePartitionsCount, ActiveControllerCount and the per-topic BrokerTopicMetrics, only in its legacy Yammer registry, which MetricsReporter does not see. When the reporter starts, it looks for Kafka’s broker class KafkaYammerMetrics. If the class is there, the reporter runs in a broker and also registers a listener on the Yammer registry. In a client, the class is absent and the Yammer side stays off, so the same JAR works in both.

The reporter also implements Kafka’s ClientTelemetry interface, the receiving end of KIP-714. On a broker, that is how metrics pushed by producers and consumers reach it.

Capture: in memory only

When Kafka calls metricChange, the reporter checks the metric’s name against the allow-list, otlp.metric.reporter.allowed.metrics, and, if it matches or the list is empty, stores a reference to the metric object in a concurrent map, keyed by metric name, and returns. The Yammer listener does the same for Yammer metrics. There is no network I/O on Kafka’s thread.

Export: one daemon thread

A single daemon thread, named monedula-metrics-reporter, runs an export every otlp.metric.reporter.interval.ms, 30 seconds by default; the first export runs one interval after the reporter starts.

An export takes a snapshot of both registries. It reads each metric’s current value once, skips values that are not finite numbers, and maps the rest to OpenTelemetry metric data, as described below. The reporter’s self-monitoring metrics join the same batch, and the whole batch goes to the collector in one call that may take at most otlp.metric.reporter.timeout.ms, 5 seconds by default. If the call fails or times out, the batch is dropped and the next export starts from a fresh snapshot; Fail-safe by design covers failure handling.

How Kafka metrics become OTLP metrics

SPI metrics whose names end in -total and Yammer counters and meters become monotonic cumulative sums, Yammer timers and histograms a _count sum plus 0.5, 0.95 and 0.99 quantile gauges, and everything else a gauge. Metric names and labels covers names, tags and scope labels.

Cumulative sums carry a start time that is fixed when the reporter starts, so a change of labels, for example when Kafka delivers the broker context after startup, never looks like a counter reset to rate().

Three pipelines, one identity

The Kafka SPI and Yammer metrics share the export thread above. Two more streams run beside it, each with its own OTLP exporter for the same collector:

• JVM runtime metrics come from OpenTelemetry’s opentelemetry-runtime-telemetry-java17 library, which reads JMX MXBeans and Java Flight Recorder streaming, through its own OpenTelemetry meter provider that exports on the same interval. otlp.metric.reporter.jvm.metrics.enabled=false turns it off.
• KIP-714 client metrics arrive on a broker request-handler thread, which copies the payload onto a bounded queue and returns. A separate daemon thread, monedula-client-telemetry, converts each payload, adds the broker and client labels, and forwards it.

The Kafka SPI, Yammer, JVM and self-monitoring metrics carry the same OTLP resource: the metrics context from Kafka, minus the namespace, plus a service.instance.id that is unique for each reporter instance, plus anything set in otlp.metric.reporter.resource.attributes, which wins on conflicts. Forwarded client metrics keep the resource the client sent, and by default the reporter adds the broker’s cluster and node IDs to it. That is why Kafka, JVM and client series join on the same broker labels in PromQL.

The connection to the collector

The reporter sends over OTLP with OpenTelemetry’s gRPC or HTTP exporter, as otlp.metric.reporter.transport selects. For HTTP, an endpoint without a path gets the standard /v1/metrics path. Headers, gzip compression, a custom CA and a client certificate for mutual TLS are set once at startup; see Secure the collector connection.

What it does not do

• Consumer lag, offset age and an Admin API based topic inventory: tools such as kafka_exporter serve those.
• Collector work: the reporter can authenticate to a collector, but fan-out, remote write, vendor exporters, batching, retry policy, credential rotation and routing belong in the OpenTelemetry Collector.
• A scrape endpoint: Prometheus support comes from the collector’s Prometheus exporter.