Kafka troubleshooting

TopicAuthorizationException (TOPIC_AUTHORIZATION_FAILED)

Kafka error · TOPIC_AUTHORIZATION_FAILED · not retriable

org.apache.kafka.common.errors.TopicAuthorizationException

The broker's authorizer found no ACL that allows this principal the operation on the topic, so it refused the request: Write for producing, Read for consuming, Describe for metadata. Once an authorizer is configured, Kafka denies by default. Find the principal the broker actually sees and add the missing ALLOW ACL with kafka-acls.sh.

Reproduce it in the playground

Scenario 10.5.1 · ACLs: authorizing a producer Seen in 1 more scenario

The moment it happensProduce denied (no ACL). t+1.000s

p1 tries to produce to orders. No ACL grants App:orders-svc the Write operation on this topic, so the broker rejects the request with TopicAuthorizationException before any record reaches the log. The producer goes to an error state and nothing is appended.

Open at this step: step 2, Produce denied (no ACL)

What it means

On a cluster with authorizer.class.name set (in KRaft, org.apache.kafka.metadata.authorizer.StandardAuthorizer), every request is checked against ACLs made of a principal, an operation, a host and a resource pattern. A resource with no matching ACL is open only to super.users, unless allow.everyone.if.no.acl.found=true, and a matching DENY ACL wins over any number of ALLOW ACLs. A failed topic check returns TOPIC_AUTHORIZATION_FAILED (error code 29), and the client throws TopicAuthorizationException with the message "Not authorized to access topics: [orders]". A producer usually logs "Topic authorization failed for topics [orders]" while fetching metadata; a consumer logs "Not authorized to read from partition orders-0." The broker records each denial at INFO in the kafka.authorizer.logger logger, for example "Principal = User:orders-svc is Denied operation = WRITE from host = 10.0.0.5 on resource = Topic:LITERAL:orders for request = Produce ... based on rule DefaultDeny". Missing rights on the consumer group or the transactional.id raise their own errors, GroupAuthorizationException and TransactionalIdAuthorizationException. Scenario 10.5.1 shows the default deny and the ALLOW Write ACL that lifts it; 10.5.2 shows the same grant through a Confluent Platform role binding, which the built-in StandardAuthorizer does not have.

Likely causes, most common first

No ACL grants the operation. Producing needs Write on the topic, consuming needs Read on the topic and on the group, and metadata or offset lookups need Describe; an ALLOW for Read, Write, Delete or Alter implies Describe. A new topic, a new service or a renamed topic has no matching ACL, so it is denied.
The principal is not the one in the ACL. With TLS client authentication the default principal is the whole certificate name, such as User:CN=orders-svc,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown, unless ssl.principal.mapping.rules shortens it; with Kerberos it is the primary part of the principal. The principal must match exactly, and the User type is case sensitive, so an ACL for User:orders-svc does not match the full certificate name unless a mapping rule produces that name.
The ACL misses the resource or the host. A literal ACL matches one exact topic name and a prefixed ACL only names that start with its prefix, so a typo or the wrong pattern type misses. An ACL limited with --allow-host matches only that client IP address, which breaks behind NAT or after a move. A matching DENY ACL overrides every ALLOW.
Authorization was switched on before the ACLs existed. Setting authorizer.class.name on a running cluster without first creating ACLs for every client denies all of them, because resources without ACLs are open only to super users.

How to fix it

Read the denial in the broker's authorizer log: it names the exact principal, operation, host and resource the broker evaluated, which settles most principal and pattern mismatches.
List every ACL that applies to the topic, wildcard and prefixed ones included: kafka-acls.sh --bootstrap-server localhost:9092 --list --topic orders --resource-pattern-type match.
Grant a client role with the convenience options: kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:orders-svc --producer --topic orders adds Write, Describe and Create on the topic; --consumer --topic orders --group my-group adds Read and Describe on the topic and Read on the group.
Or grant a single operation, for example --add --allow-principal User:orders-svc --operation Write --topic orders, with --resource-pattern-type prefixed to cover a family of topics.
Retry once the ACL exists: a consumer and a non-transactional producer carry on without a restart. A transactional producer must at least abort the open transaction; the KafkaProducer Javadoc says to close it, which is the safe choice.

What not to do

Setting allow.everyone.if.no.acl.found=true, adding a User:* ACL or making every service a super user to make the error go away: each one gives access far beyond the topic that failed.

Does the client retry it?

No. The same principal is denied every time until an ACL allows the operation. Once one does, a consumer and a non-transactional producer carry on without a restart. A transactional producer must at least abort the open transaction; the KafkaProducer Javadoc says to close it, which is the safe choice.

Configs involved

authorizer.class.name The class brokers and controllers use to check each request against ACLs; empty by default, so nothing is checked, and org.apache.kafka.metadata.authorizer.StandardAuthorizer on a KRaft cluster.
super.users Principals the authorizer allows every operation whatever the ACLs say, as a semicolon-separated list such as User:admin;User:broker.
allow.everyone.if.no.acl.found A StandardAuthorizer setting: when true, a resource that no ACL matches is open to every principal; when false (the default), only super.users may use it.
ssl.principal.mapping.rules Rules that turn the distinguished name of a client TLS certificate into the user name ACLs match; by default the whole distinguished name is the principal.
sasl.kerberos.principal.to.local.rules Rules that turn a Kerberos principal into the short user name ACLs match; by default user/host@REALM becomes user.

Reproduce it in the simulator

TopicAuthorizationException: authorizing a Kafka producer (ACLs: authorizing a producer) Open at the moment it happens: TopicAuthorizationException: authorizing a Kafka producer
Confluent RBAC role bindings for Kafka topics (RBAC: role bindings) Open at the moment it happens: Confluent RBAC role bindings for Kafka topics

Sources

All troubleshooting pages