Monedula
monedula-metrics-reporter docs menu

Collect Kafka client metrics with KIP-714

Forward KIP-714 client telemetry through the broker without touching client apps: create a client-metrics subscription and pick labels.

How-to guides 3 of 6

4 min read

Updated 10 October 2026

Markdown

On this page

With KIP-714, producers and consumers push their own metrics to the broker: client-side request latency, record queue time, rebalance and coordinator timings, connection churn. The reporter on your brokers receives that telemetry and forwards it to your collector, next to the broker’s own metrics, with no change to the client applications.

Before you start

  • The reporter installed on every broker, with otlp.metric.reporter.client.telemetry.enabled left at its default, true. See Install the reporter on brokers.
  • Clients that push KIP-714 metrics, such as the Java client or librdkafka 2.5 or later; see Compatibility.
  • Kafka’s kafka-client-metrics.sh tool, run against your cluster.

Steps

1. Create a client-metrics subscription

A subscription tells clients which metrics to push and how often; the receiver stays idle until one exists. This one asks for all metrics every 5 seconds:

kafka-client-metrics.sh --bootstrap-server localhost:9092 --alter \
  --name all-clients --metrics "*" --interval 5000

The subscription is a CLIENT_METRICS dynamic config, so kafka-configs.sh --entity-type client-metrics or the Admin client’s incrementalAlterConfigs can create it too. Clients fetch the subscription and start pushing; allow about a minute before their metrics appear. The subscription decides which client metrics are collected; the reporter’s allowed.metrics filter does not apply to them. The broker keeps the subscription in cluster metadata; delete it with --delete --name all-clients to stop the pushes.

2. Choose the labels

Forwarded client metrics keep the labels the client sends, and the reporter adds labels from the broker’s side of the connection. Four keys decide which:

Key Default Adds
otlp.metric.reporter.client.telemetry.enrich.broker true kafka_cluster_id and kafka_node_id of the receiving broker, so client and broker series join in PromQL
otlp.metric.reporter.client.telemetry.enrich.client.id true client_id, so metrics of different applications stay apart. Moderate cardinality
otlp.metric.reporter.client.telemetry.enrich.client.instance.id false client_instance_id, the KIP-714 UUID of each client instance. High cardinality
otlp.metric.reporter.client.telemetry.enrich.client.identity false kafka_client_principal and kafka_client_address, the authenticated principal and the client’s address. High cardinality and personal data

The broker labels are applied after the client’s own attributes, so a client cannot spoof kafka_cluster_id. Turn the opt-in labels on in server.properties:

otlp.metric.reporter.client.telemetry.enrich.client.instance.id=true

Restart each broker to apply the change.

Check it worked

The reporter’s self-monitoring metrics show the forwarding:

  • monedula_reporter_clienttelemetry_received_total and monedula_reporter_clienttelemetry_forwarded_total grow.
  • monedula_reporter_clienttelemetry_dropped_total stays flat. If it grows, payloads hit a full queue or a failed export.

In Prometheus, client metrics keep the names the client pushes, such as org_apache_kafka_producer_request_latency_avg for a Java producer’s request latency, with the default client_id, kafka_cluster_id and kafka_node_id labels.

In builds from main: change the labels at runtime

Builds from the main branch, not yet released, let the four enrich keys change on running brokers through Kafka’s dynamic broker configuration, with the same kafka-configs.sh command that changes the allow-list. This does not work with 0.10.0.

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default \
  --alter --add-config 'otlp.metric.reporter.client.telemetry.enrich.client.instance.id=true'

The change applies from the next client payload the broker forwards. --entity-default changes every broker; --entity-name <broker-id> changes one. An invalid value, such as a toggle that is neither true nor false, fails the command and leaves the running reporter untouched.

One exception: the broker captures a client’s principal and address only when the reporter starts with enrich.client.identity=true. Turning that key on at runtime, after the broker started with false, adds no labels until the next restart. Turning it off at runtime takes effect from the next forwarded payload.

Limits

  • Gauges, sums and histograms are forwarded. Other OTLP types, exponential histograms and summaries, are dropped and counted in monedula_reporter_clienttelemetry_unsupported_metrics_dropped_total. The standard KIP-714 client metrics fit within these types.
  • Pushes wait in a bounded in-memory queue; when it is full, new payloads are dropped and counted, and the broker’s request handling never waits (Fail-safe by design).
  • Kafka hands the reporter the broker’s metrics context during broker startup. Client metrics forwarded before that carry no kafka_cluster_id or kafka_node_id labels.
  • Kafka uses one client-telemetry receiver per broker; see Compatibility for other telemetry plugins.
  • There are no client software labels. KIP-714 defines client_software_name and client_software_version, but Kafka does not pass them to the receiver. The instrumentation scope, otel_scope_name and otel_scope_version, is the closest signal: librdkafka reports its client ID and version there, and the Java client a generic kafka-client scope.
  • Client libraries push different metric sets. Every compliant client pushes a small standard set; the Java client pushes its full internal registry on top, more than 100 series, while librdkafka-based clients, such as confluent-kafka for Python, Go, .NET and Node, push only the standard set.

To stop the broker from advertising client telemetry at all, set otlp.metric.reporter.client.telemetry.enabled=false and restart.