Fail-safe by design

The reporter runs inside the Kafka process, so it trades metrics completeness for Kafka availability: a collector that is unreachable, slow or misconfigured never holds up Kafka’s request, produce or fetch paths.

Kafka’s threads never wait for the reporter

Kafka calls metricChange and metricRemoval on its own threads, often on hot paths. Those callbacks only put or remove a reference in an in-memory map, after checking the name against the allow-list once per registration, and the KIP-714 receiver only offers the payload bytes to a bounded queue without waiting. Parsing, conversion and every network call run later, on the reporter’s own daemon threads; How it works describes them.

A failed export drops the batch

Each export waits at most otlp.metric.reporter.timeout.ms for the collector. If the call fails or times out, the reporter logs a warning, counts the failure in monedula_reporter_export_failure_total, drops the batch and waits for the next interval. The next export takes a fresh snapshot of the latest values. Nothing is queued for a retry.

The timeout must be shorter than the export interval, because a slow export could otherwise run into the next tick on the single export thread. An unexpected error during an export, even a Java Error, is caught and logged, and the next tick runs as scheduled, so one bad tick never stops exports for good.

What you trade is the data points of the intervals that the collector missed. Counters are cumulative since the reporter started, so the next successful export still carries the full totals: the events of the missed intervals keep their count and lose only their timing.

Memory stays bounded

The registries store references to the metric objects Kafka already holds, one per metric, not copies and not a history of values. Their size is bounded by the number of metrics the broker announces. Internal caches of names and labels are pruned to the live set of metrics on every export, so they do not grow when topics come and go.

Client telemetry waits in a queue whose size is fixed by otlp.metric.reporter.client.telemetry.queue.capacity, 1024 payloads by default. When the queue is full, a new payload is dropped and counted in monedula_reporter_clienttelemetry_dropped_total; the request-handler thread does not wait. A payload that fails to export is dropped and counted the same way.

A bad configuration degrades to no-op

Kafka rethrows an exception from a reporter’s configure method, which would fail the broker’s startup. The reporter catches every configuration problem itself: an invalid allow-list pattern, a malformed endpoint, a header without a value, a missing or unreadable certificate file, a client certificate without its key, a timeout not shorter than the interval, a toggle that is neither true nor false. It logs OtlpMetricReporter failed to initialize, running as no-op at ERROR and stays inactive for the life of the JVM, while Kafka serves traffic as usual.

Optional parts fail on their own:

• If the JVM metrics cannot start, the reporter logs a warning and continues without them. If only Java Flight Recorder cannot start, for example with a read-only temporary directory in a slim container, the JVM metrics fall back to JMX alone.
• If client telemetry cannot start, only client telemetry is disabled; the Kafka and JVM metrics keep flowing.
• If Kafka’s metrics context cannot be applied, the reporter continues without the context labels.

Shutdown is bounded

When Kafka closes the reporter, each step of its shutdown, stopping the export threads and closing the exporters, waits at most otlp.metric.reporter.timeout.ms before it moves on, so a slow collector cannot hold up a broker shutdown indefinitely.

Watching the reporter itself

A failing pipeline does not announce itself through Kafka. The reporter’s self-monitoring metrics count successful and failed exports, measure how long they take and describe the client telemetry queue, and the example alert rules turn them into alerts.