--- title: "Metric names and labels" description: "How Kafka metric names become OTLP and Prometheus names, and which broker, topic and partition labels every series carries." url: https://monedula.dev/flock/docs/metrics-reporter/concepts/metric-names-and-labels/ --- The reporter turns the names from Kafka's two registries, neither of which Prometheus accepts as they are, into one lowercase, Prometheus-friendly form, Strimzi-style. Kafka's tags, Yammer scopes and the broker's identity become labels. ## Flattening rules Every name is lowercased, every character other than a letter or a digit becomes an underscore, consecutive underscores collapse into one, and leading and trailing underscores are removed. What goes into the name depends on the registry. ### Kafka SPI metrics An SPI metric is named `{namespace}_{group}_{name}`. The namespace comes from Kafka's metrics context and depends on the process: `kafka.server` on a broker, `kafka.producer` or `kafka.consumer` in a client. On a broker, the `socket-server-metrics` group's `connection-count` metric becomes: ```text kafka_server_socket_server_metrics_connection_count ``` If the context has no namespace, the prefix is left out: `socket_server_metrics_connection_count`. Kafka's metric tags become labels of the series. ### Yammer metrics A Yammer metric is named `{group}_{type}_{name}`, without a namespace. The broker's `kafka.server` group, `ReplicaManager` type and `UnderReplicatedPartitions` name become `kafka_server_replicamanager_underreplicatedpartitions`. The Yammer scope becomes labels. Kafka encodes it as `key.value.key.value`: - `topic.` becomes a `topic` label. - `topic..partition.` becomes `topic` and `partition` labels. - Any other scope is split pairwise into labels. A scope that does not split into pairs becomes a single `scope` label. ```text # Per-topic (scope="topic.") kafka_server_brokertopicmetrics_bytesinpersec_total{topic="my-topic"} # Per-partition (scope="topic..partition.") kafka_cluster_partition_underreplicated{topic="my-topic", partition="0"} ``` Some metrics exist at both levels: `kafka_server_replicamanager_underreplicatedpartitions` is the broker-wide aggregate, without `topic` or `partition` labels, and `kafka_cluster_partition_underreplicated{topic, partition}` the per-partition gauge that shows which partition is degraded. Timers and histograms become a `_count` counter plus one gauge series per quantile, 0.5, 0.95 and 0.99, told apart by a `quantile` label. ## Counters and gauges Kafka SPI metrics whose names end in `-total` are cumulative counters and are exported as monotonic sums; all other SPI metrics are gauges. Yammer counters and meters are exported as monotonic sums of their count, and Yammer gauges as gauges. On the way into Prometheus, the collector's Prometheus exporter adds a `_total` suffix to monotonic sums whose names do not end in `_total` already. The Yammer meter `BytesInPerSec` leaves the reporter as `kafka_server_brokertopicmetrics_bytesinpersec` and is queried as `kafka_server_brokertopicmetrics_bytesinpersec_total`, as in the quickstart dashboards. Counters are cumulative since the reporter started and never reset while the JVM runs. Query them with `rate()` or `increase()`, never as raw values. ## Labels on every series Kafka gives a broker's reporter a metrics context with the cluster ID and the node ID. The reporter attaches them to every series as OTLP resource attributes, and the collector's Prometheus exporter turns them into labels when `resource_to_telemetry_conversion` is enabled. The dots in the attribute names become underscores: ```text kafka_controller_kafkacontroller_activecontrollercount{ kafka_cluster_id="...", kafka_node_id="1" } 1 ``` - `kafka_cluster_id`: the cluster ID. - `kafka_node_id` on KRaft brokers, or `kafka_broker_id` on ZooKeeper-mode brokers. - `service_instance_id`: a UUID generated for each reporter instance, so two reporters in the same JVM stay distinguishable. A restarted reporter gets a new one. Override it with a stable value through `otlp.metric.reporter.resource.attributes=service.instance.id=...`. The namespace from the context is not a label; it is the prefix of SPI metric names. Attributes set in `otlp.metric.reporter.resource.attributes`, as `key=value,key=value`, are added to every series and win over Kafka's context when a key appears in both. In a producer or consumer, Kafka's context has no cluster or node ID, so client series do not carry those labels. JVM runtime metrics carry the same resource labels as the Kafka metrics, so `by (kafka_cluster_id, kafka_node_id)` joins them with broker series. ## JVM, self-monitoring and client metrics These streams keep the names their sources give them: - **JVM runtime metrics** use the names of OpenTelemetry's runtime instrumentation, such as `jvm_memory_used_bytes`. - **Self-monitoring metrics** start with `monedula_reporter_`; the [self-monitoring reference](/flock/docs/metrics-reporter/reference/self-monitoring-metrics/) lists them. - **KIP-714 client metrics** keep the names the client pushes, such as `org_apache_kafka_producer_request_latency_avg` for the Java producer's request latency in Prometheus. The reporter adds `kafka_cluster_id`, `kafka_node_id` and `client_id` by default; [Collect Kafka client metrics with KIP-714](/flock/docs/metrics-reporter/how-to/collect-client-telemetry/) lists the optional labels. The client's instrumentation scope arrives as `otel_scope_name` and `otel_scope_version`. ## Sample names Once the reporter runs on brokers and clients, these names, among many others, appear in Prometheus: ```text # Broker-side Kafka SPI kafka_server_socket_server_metrics_connection_count # Broker-side Yammer kafka_controller_kafkacontroller_activecontrollercount kafka_controller_kafkacontroller_offlinepartitionscount kafka_server_replicamanager_underreplicatedpartitions kafka_server_brokertopicmetrics_bytesinpersec_total kafka_network_requestmetrics_requestspersec_total kafka_cluster_partition_underreplicated # Java producer with the reporter, Kafka SPI kafka_producer_producer_metrics_record_send_rate # JVM runtime (memory, GC, threads, classes, CPU) jvm_thread_count jvm_memory_used_bytes jvm_gc_duration_seconds_sum jvm_cpu_time_seconds_total # Reporter self-monitoring monedula_reporter_export_success_total monedula_reporter_export_failure_total monedula_reporter_export_duration_ms ```