--- title: "How it works" description: "How monedula-metrics-reporter hooks into Kafka's MetricsReporter interface and the Yammer registry, batches metrics and exports them over OTLP." url: https://monedula.dev/flock/docs/metrics-reporter/concepts/how-it-works/ --- Kafka's threads only record metric references in memory; the reporter's own threads do all export I/O. ```text Kafka MetricsReporter SPI -> MetricRegistry --+ Kafka Yammer registry -> YammerMetricRegistry --+-> export thread -> OTLP exporter -> collector Kafka MetricsContext -> resource attributes --+ ``` ## Where it plugs in Kafka loads the reporter through `metric.reporters`, as an implementation of its `MetricsReporter` interface. Kafka then calls it at a few points: - `init` and `metricChange` when metrics are registered or updated, and `metricRemoval` when they go away. - `contextChange` with the process's metrics context: on a broker, the cluster ID and the node ID (or broker ID in ZooKeeper mode), and the namespace that becomes the prefix of metric names. - `configure` and `close` for its settings and lifecycle. Kafka keeps several broker-internal metrics, such as `UnderReplicatedPartitions`, `OfflinePartitionsCount`, `ActiveControllerCount` and the per-topic `BrokerTopicMetrics`, only in its legacy Yammer registry, which `MetricsReporter` does not see. When the reporter starts, it looks for Kafka's broker class `KafkaYammerMetrics`. If the class is there, the reporter runs in a broker and also registers a listener on the Yammer registry. In a client, the class is absent and the Yammer side stays off, so the same JAR works in both. The reporter also implements Kafka's `ClientTelemetry` interface, the receiving end of [KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability). On a broker, that is how metrics pushed by producers and consumers reach it. ## Capture: in memory only When Kafka calls `metricChange`, the reporter checks the metric's name against the allow-list, `otlp.metric.reporter.allowed.metrics`, and, if it matches or the list is empty, stores a reference to the metric object in a concurrent map, keyed by metric name, and returns. The Yammer listener does the same for Yammer metrics. There is no network I/O on Kafka's thread. ## Export: one daemon thread A single daemon thread, named `monedula-metrics-reporter`, runs an export every `otlp.metric.reporter.interval.ms`, 30 seconds by default; the first export runs one interval after the reporter starts. An export takes a snapshot of both registries. It reads each metric's current value once, skips values that are not finite numbers, and maps the rest to OpenTelemetry metric data, as described below. The reporter's [self-monitoring metrics](/flock/docs/metrics-reporter/reference/self-monitoring-metrics/) join the same batch, and the whole batch goes to the collector in one call that may take at most `otlp.metric.reporter.timeout.ms`, 5 seconds by default. If the call fails or times out, the batch is dropped and the next export starts from a fresh snapshot; [Fail-safe by design](/flock/docs/metrics-reporter/concepts/fail-safe-design/) covers failure handling. ## How Kafka metrics become OTLP metrics SPI metrics whose names end in `-total` and Yammer counters and meters become monotonic cumulative sums, Yammer timers and histograms a `_count` sum plus 0.5, 0.95 and 0.99 quantile gauges, and everything else a gauge. [Metric names and labels](/flock/docs/metrics-reporter/concepts/metric-names-and-labels/) covers names, tags and scope labels. Cumulative sums carry a start time that is fixed when the reporter starts, so a change of labels, for example when Kafka delivers the broker context after startup, never looks like a counter reset to `rate()`. ## Three pipelines, one identity The Kafka SPI and Yammer metrics share the export thread above. Two more streams run beside it, each with its own OTLP exporter for the same collector: - **JVM runtime metrics** come from OpenTelemetry's `opentelemetry-runtime-telemetry-java17` library, which reads JMX MXBeans and Java Flight Recorder streaming, through its own OpenTelemetry meter provider that exports on the same interval. `otlp.metric.reporter.jvm.metrics.enabled=false` turns it off. - **KIP-714 client metrics** arrive on a broker request-handler thread, which copies the payload onto a bounded queue and returns. A separate daemon thread, `monedula-client-telemetry`, converts each payload, adds the broker and client labels, and forwards it. The Kafka SPI, Yammer, JVM and self-monitoring metrics carry the same OTLP resource: the metrics context from Kafka, minus the namespace, plus a `service.instance.id` that is unique for each reporter instance, plus anything set in `otlp.metric.reporter.resource.attributes`, which wins on conflicts. Forwarded client metrics keep the resource the client sent, and by default the reporter adds the broker's cluster and node IDs to it. That is why Kafka, JVM and client series join on the same broker labels in PromQL. ## The connection to the collector The reporter sends over OTLP with OpenTelemetry's gRPC or HTTP exporter, as `otlp.metric.reporter.transport` selects. For HTTP, an endpoint without a path gets the standard `/v1/metrics` path. Headers, gzip compression, a custom CA and a client certificate for mutual TLS are set once at startup; see [Secure the collector connection](/flock/docs/metrics-reporter/how-to/secure-the-collector-connection/). ## What it does not do - Consumer lag, offset age and an Admin API based topic inventory: tools such as `kafka_exporter` serve those. - Collector work: the reporter can authenticate to a collector, but fan-out, remote write, vendor exporters, batching, retry policy, credential rotation and routing belong in the OpenTelemetry Collector. - A scrape endpoint: Prometheus support comes from the collector's Prometheus exporter.