Self-monitoring metrics

The reporter emits metrics about its own health in every export batch, next to the Kafka metrics and with the same resource labels. The names below are as they appear in Prometheus.

Export metrics

Emitted by every reporter instance, on brokers and clients.

Metric	Type	Meaning
monedula_reporter_export_success_total	counter	Successful OTLP exports since the reporter started.
monedula_reporter_export_failure_total	counter	Failed exports since the reporter started: timeouts, refused connections, mapping errors.
monedula_reporter_export_duration_ms	gauge	Wall-clock duration of the previous export tick, in milliseconds. Spikes point to a slow collector.

The values in a batch describe the exports before it: an export’s own success or failure is counted after it is sent, so it shows up in the next batch.

Client telemetry metrics

Emitted while KIP-714 client telemetry is enabled, otlp.metric.reporter.client.telemetry.enabled=true, which is the default.

Metric	Type	Meaning
monedula_reporter_clienttelemetry_received_total	counter	Client telemetry payloads accepted, one per push.
monedula_reporter_clienttelemetry_forwarded_total	counter	Payloads forwarded to the collector.
monedula_reporter_clienttelemetry_dropped_total	counter	Payloads dropped because the queue was full or the export failed.
monedula_reporter_clienttelemetry_unsupported_metrics_dropped_total	counter	Individual data points dropped because their OTLP type is not supported.
monedula_reporter_clienttelemetry_queue_depth	gauge	Current number of payloads in the inbound queue.

The received, forwarded and dropped counters count whole payloads. The unsupported counter counts single data points within a payload and does not mark the payload as dropped.

Counters, labels and filtering

The counters are cumulative and start again from zero when the broker or client restarts. The metrics carry the reporter’s resource labels: kafka_cluster_id and kafka_node_id, or kafka_broker_id, on a broker, and service_instance_id everywhere. Metric names and labels covers querying counters with rate() and setting a stable service_instance_id.

The reporter adds these metrics to every batch itself, so otlp.metric.reporter.allowed.metrics does not apply to them.

Example alert

rate(monedula_reporter_export_failure_total[5m]) > 0

Alert on broker health covers the complete example rules, including an alert for when these metrics stop arriving altogether.

Before 0.10.0

In 0.9.0, the duration metric reached Prometheus as monedula_reporter_export_duration_ms_milliseconds. Dashboards, recording rules and alerts built against that name need the documented name, monedula_reporter_export_duration_ms, after an upgrade.