Monedula
monedula-metrics-reporter docs menu

Alert on broker health

Load the example Prometheus alert rules from the quickstart to alert on under-replicated and offline partitions, the controller and the reporter itself.

How-to guides 6 of 6

3 min read

Updated 5 October 2026

Markdown

On this page

The quickstart ships an example rules file, quickstart/prometheus-alerts.yml, with seven Prometheus alerts: four on the reporter’s own health and three on Kafka broker health. The quickstart stack does not load it by default; it is a reference for wiring up production.

Before you start

  • The reporter’s metrics in Prometheus with the kafka_cluster_id and kafka_node_id labels, which the rules use. Step 5 of Install the reporter on brokers sets them up.
  • A Prometheus server whose configuration you can change.

Steps

1. Copy the rules file

Copy prometheus-alerts.yml to your Prometheus server, for example as /etc/prometheus/alerts.yml.

2. Add it to rule_files

Reference the file in your Prometheus configuration’s rule_files list, then reload or restart Prometheus:

rule_files:
  - /etc/prometheus/alerts.yml

To try the rules in the quickstart first, add the same rule_files entry to quickstart/prometheus.yml. Then mount the rules file in the prometheus service of quickstart/docker-compose.yml:

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus-alerts.yml:/etc/prometheus/alerts.yml

Run docker compose up -d in the quickstart directory to recreate Prometheus with the rules.

3. Tune the rules for your cluster

The broker alerts are a small starter set: tune their thresholds to your cluster’s normal operating range before you rely on them in production. The alerts’ summaries and descriptions name brokers by kafka_node_id, which KRaft brokers carry; for ZooKeeper-mode brokers, which carry kafka_broker_id, change the annotations.

The alerts

Reporter health

These alerts watch the reporter’s self-monitoring metrics, which the reporter emits on every export.

Alert Expression For Severity
ReporterExportFailing rate(monedula_reporter_export_failure_total[5m]) > 0 10m warning
ReporterNoSuccessfulExports rate(monedula_reporter_export_success_total[5m]) == 0 and on(service_instance_id) rate(monedula_reporter_export_failure_total[5m]) > 0 5m critical
ReporterExportDurationSpiking monedula_reporter_export_duration_ms > 1000 15m warning
ReporterMetricsAbsent absent(monedula_reporter_export_success_total) 10m critical
  • ReporterExportFailing: Kafka traffic is unaffected, but the metrics pipeline is not. Check the collector endpoint, network reachability and the broker log for OTLP metric export failed or timed out.
  • ReporterNoSuccessfulExports: every export attempt of a reporter instance fails, so none of its metrics reach the collector.
  • ReporterExportDurationSpiking: the collector is likely backpressured or the network is slow. Sustained values that approach otlp.metric.reporter.timeout.ms mean that batches are about to be dropped.
  • ReporterMetricsAbsent: the reporter or its JVM is down, or the pipeline to Prometheus is broken. The self-monitoring metrics travel through the same pipeline as everything else, so this is the alert that catches a total outage.

Kafka broker health

Alert Expression For Severity
KafkaUnderReplicatedPartitions kafka_server_replicamanager_underreplicatedpartitions > 0 5m warning
KafkaOfflinePartitions kafka_controller_kafkacontroller_offlinepartitionscount > 0 1m critical
KafkaNoActiveController sum by (kafka_cluster_id) (kafka_controller_kafkacontroller_activecontrollercount) != 1 1m critical

KafkaNoActiveController fires when the active-controller counts across the cluster do not add up to exactly 1: during a controller election, or when no node is acting as controller.

Check it worked

Open the Alerts page of the Prometheus UI. It lists the groups monedula-reporter-health and kafka-broker-health with their rules, each inactive, pending or firing. On a healthy cluster they stay inactive.