---
title: "Collect Kafka client metrics with KIP-714"
description: "Forward KIP-714 client telemetry through the broker without touching client apps: create a client-metrics subscription and pick labels."
url: https://monedula.dev/flock/docs/metrics-reporter/how-to/collect-client-telemetry/
---

With [KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability), producers and consumers push their own metrics to the broker: client-side request latency, record queue time, rebalance and coordinator timings, connection churn. The reporter on your brokers receives that telemetry and forwards it to your collector, next to the broker's own metrics, with no change to the client applications.

## Before you start

- The reporter installed on every broker, with `otlp.metric.reporter.client.telemetry.enabled` left at its default, `true`. See [Install the reporter on brokers](/flock/docs/metrics-reporter/how-to/install-on-brokers/).
- Clients that push KIP-714 metrics, such as the Java client or librdkafka 2.5 or later; see [Compatibility](/flock/docs/metrics-reporter/concepts/compatibility/).
- Kafka's `kafka-client-metrics.sh` tool, run against your cluster.

## Steps

### 1. Create a client-metrics subscription

A subscription tells clients which metrics to push and how often; the receiver stays idle until one exists. This one asks for all metrics every 5 seconds:

```bash
kafka-client-metrics.sh --bootstrap-server localhost:9092 --alter \
  --name all-clients --metrics "*" --interval 5000
```

The subscription is a `CLIENT_METRICS` dynamic config, so `kafka-configs.sh --entity-type client-metrics` or the Admin client's `incrementalAlterConfigs` can create it too. Clients fetch the subscription and start pushing; allow about a minute before their metrics appear. The subscription decides which client metrics are collected; the reporter's `allowed.metrics` filter does not apply to them. The broker keeps the subscription in cluster metadata; delete it with `--delete --name all-clients` to stop the pushes.

### 2. Choose the labels

Forwarded client metrics keep the labels the client sends, and the reporter adds labels from the broker's side of the connection. Four keys decide which:

| Key | Default | Adds |
| --- | --- | --- |
| `otlp.metric.reporter.client.telemetry.enrich.broker` | `true` | `kafka_cluster_id` and `kafka_node_id` of the receiving broker, so client and broker series join in PromQL |
| `otlp.metric.reporter.client.telemetry.enrich.client.id` | `true` | `client_id`, so metrics of different applications stay apart. Moderate cardinality |
| `otlp.metric.reporter.client.telemetry.enrich.client.instance.id` | `false` | `client_instance_id`, the KIP-714 UUID of each client instance. High cardinality |
| `otlp.metric.reporter.client.telemetry.enrich.client.identity` | `false` | `kafka_client_principal` and `kafka_client_address`, the authenticated principal and the client's address. High cardinality and personal data |

The broker labels are applied after the client's own attributes, so a client cannot spoof `kafka_cluster_id`. Turn the opt-in labels on in `server.properties`:

```properties
otlp.metric.reporter.client.telemetry.enrich.client.instance.id=true
```

Restart each broker to apply the change.

## Check it worked

The reporter's [self-monitoring metrics](/flock/docs/metrics-reporter/reference/self-monitoring-metrics/) show the forwarding:

- `monedula_reporter_clienttelemetry_received_total` and `monedula_reporter_clienttelemetry_forwarded_total` grow.
- `monedula_reporter_clienttelemetry_dropped_total` stays flat. If it grows, payloads hit a full queue or a failed export.

In Prometheus, client metrics keep the names the client pushes, such as `org_apache_kafka_producer_request_latency_avg` for a Java producer's request latency, with the default `client_id`, `kafka_cluster_id` and `kafka_node_id` labels.

## In builds from main: change the labels at runtime

Builds from the main branch, not yet released, let the four `enrich` keys change on running brokers through Kafka's dynamic broker configuration, with the same `kafka-configs.sh` command that [changes the allow-list](/flock/docs/metrics-reporter/how-to/filter-metrics/). This does not work with 0.10.0.

```bash
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default \
  --alter --add-config 'otlp.metric.reporter.client.telemetry.enrich.client.instance.id=true'
```

The change applies from the next client payload the broker forwards. `--entity-default` changes every broker; `--entity-name <broker-id>` changes one. An invalid value, such as a toggle that is neither `true` nor `false`, fails the command and leaves the running reporter untouched.

One exception: the broker captures a client's principal and address only when the reporter starts with `enrich.client.identity=true`. Turning that key on at runtime, after the broker started with `false`, adds no labels until the next restart. Turning it off at runtime takes effect from the next forwarded payload.

## Limits

- Gauges, sums and histograms are forwarded. Other OTLP types, exponential histograms and summaries, are dropped and counted in `monedula_reporter_clienttelemetry_unsupported_metrics_dropped_total`. The standard KIP-714 client metrics fit within these types.
- Pushes wait in a bounded in-memory queue; when it is full, new payloads are dropped and counted, and the broker's request handling never waits ([Fail-safe by design](/flock/docs/metrics-reporter/concepts/fail-safe-design/#memory-stays-bounded)).
- Kafka hands the reporter the broker's metrics context during broker startup. Client metrics forwarded before that carry no `kafka_cluster_id` or `kafka_node_id` labels.
- Kafka uses one client-telemetry receiver per broker; see [Compatibility](/flock/docs/metrics-reporter/concepts/compatibility/#next-to-other-plugins) for other telemetry plugins.
- There are no client software labels. KIP-714 defines `client_software_name` and `client_software_version`, but Kafka does not pass them to the receiver. The instrumentation scope, `otel_scope_name` and `otel_scope_version`, is the closest signal: librdkafka reports its client ID and version there, and the Java client a generic `kafka-client` scope.
- Client libraries push different metric sets. Every compliant client pushes a small standard set; the Java client pushes its full internal registry on top, more than 100 series, while librdkafka-based clients, such as confluent-kafka for Python, Go, .NET and Node, push only the standard set.

To stop the broker from advertising client telemetry at all, set `otlp.metric.reporter.client.telemetry.enabled=false` and restart.

## Related

- [Report metrics from Java clients](/flock/docs/metrics-reporter/how-to/report-client-metrics/) runs the reporter inside a Java client instead.
- [Configuration keys](/flock/docs/metrics-reporter/reference/configuration/) lists the `client.telemetry` keys with their defaults.
- Runnable version: [quickstart](https://github.com/monedula-dev/monedula-metrics-reporter/tree/main/quickstart), whose demo service creates a subscription and runs a producer and a consumer that push their metrics.