---
title: "Alert on broker health"
description: "Load the example Prometheus alert rules from the quickstart to alert on under-replicated and offline partitions, the controller and the reporter itself."
url: https://monedula.dev/flock/docs/metrics-reporter/how-to/alert-on-broker-health/
---
The quickstart ships an example rules file, [`quickstart/prometheus-alerts.yml`](https://github.com/monedula-dev/monedula-metrics-reporter/blob/main/quickstart/prometheus-alerts.yml), with seven Prometheus alerts: four on the reporter's own health and three on Kafka broker health. The quickstart stack does not load it by default; it is a reference for wiring up production.

## Before you start

- The reporter's metrics in Prometheus with the `kafka_cluster_id` and `kafka_node_id` labels, which the rules use. Step 5 of [Install the reporter on brokers](/flock/docs/metrics-reporter/how-to/install-on-brokers/) sets them up.
- A Prometheus server whose configuration you can change.

## Steps

### 1. Copy the rules file

Copy [`prometheus-alerts.yml`](https://github.com/monedula-dev/monedula-metrics-reporter/blob/main/quickstart/prometheus-alerts.yml) to your Prometheus server, for example as `/etc/prometheus/alerts.yml`.

### 2. Add it to rule_files

Reference the file in your Prometheus configuration's `rule_files` list, then reload or restart Prometheus:

```yaml
rule_files:
  - /etc/prometheus/alerts.yml
```

To try the rules in the quickstart first, add the same `rule_files` entry to `quickstart/prometheus.yml`. Then mount the rules file in the `prometheus` service of `quickstart/docker-compose.yml`:

```yaml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus-alerts.yml:/etc/prometheus/alerts.yml
```

Run `docker compose up -d` in the `quickstart` directory to recreate Prometheus with the rules.

### 3. Tune the rules for your cluster

The broker alerts are a small starter set: tune their thresholds to your cluster's normal operating range before you rely on them in production. The alerts' summaries and descriptions name brokers by `kafka_node_id`, which KRaft brokers carry; for ZooKeeper-mode brokers, which carry `kafka_broker_id`, change the annotations.

## The alerts

### Reporter health

These alerts watch the reporter's [self-monitoring metrics](/flock/docs/metrics-reporter/reference/self-monitoring-metrics/), which the reporter emits on every export.

| Alert | Expression | For | Severity |
| --- | --- | --- | --- |
| `ReporterExportFailing` | `rate(monedula_reporter_export_failure_total[5m]) > 0` | 10m | warning |
| `ReporterNoSuccessfulExports` | `rate(monedula_reporter_export_success_total[5m]) == 0 and on(service_instance_id) rate(monedula_reporter_export_failure_total[5m]) > 0` | 5m | critical |
| `ReporterExportDurationSpiking` | `monedula_reporter_export_duration_ms > 1000` | 15m | warning |
| `ReporterMetricsAbsent` | `absent(monedula_reporter_export_success_total)` | 10m | critical |

- **ReporterExportFailing**: Kafka traffic is unaffected, but the metrics pipeline is not. Check the collector endpoint, network reachability and the broker log for `OTLP metric export failed or timed out`.
- **ReporterNoSuccessfulExports**: every export attempt of a reporter instance fails, so none of its metrics reach the collector.
- **ReporterExportDurationSpiking**: the collector is likely backpressured or the network is slow. Sustained values that approach `otlp.metric.reporter.timeout.ms` mean that batches are about to be dropped.
- **ReporterMetricsAbsent**: the reporter or its JVM is down, or the pipeline to Prometheus is broken. The self-monitoring metrics travel through the same pipeline as everything else, so this is the alert that catches a total outage.

### Kafka broker health

| Alert | Expression | For | Severity |
| --- | --- | --- | --- |
| `KafkaUnderReplicatedPartitions` | `kafka_server_replicamanager_underreplicatedpartitions > 0` | 5m | warning |
| `KafkaOfflinePartitions` | `kafka_controller_kafkacontroller_offlinepartitionscount > 0` | 1m | critical |
| `KafkaNoActiveController` | `sum by (kafka_cluster_id) (kafka_controller_kafkacontroller_activecontrollercount) != 1` | 1m | critical |

`KafkaNoActiveController` fires when the active-controller counts across the cluster do not add up to exactly 1: during a controller election, or when no node is acting as controller.

## Check it worked

Open the **Alerts** page of the Prometheus UI. It lists the groups `monedula-reporter-health` and `kafka-broker-health` with their rules, each inactive, pending or firing. On a healthy cluster they stay inactive.

## Related

- [Metric names and labels](/flock/docs/metrics-reporter/concepts/metric-names-and-labels/) explains the broker metrics and their labels.
- [Prometheus alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) documents the rule format.
- Runnable version: [quickstart](https://github.com/monedula-dev/monedula-metrics-reporter/tree/main/quickstart), which ships the rules file next to its Prometheus configuration.