Check connectivity with doctor

doctor never changes anything, so run it before the first apply against a new cluster, or as the first step of a pipeline. It takes no manifests and checks one cluster at a time.

Before you start

• A cluster config file, clusters/prod.yaml on this page, and the environment variables and files its valueFrom references point to. Connect to a secured Kafka cluster shows how to write one.

Steps

1. Run doctor

monedula-gitops doctor --cluster-config-file ./clusters/prod.yaml

When the config file or directory defines more than one cluster, select one with --cluster. -o json or -o yaml returns the result as a document:

monedula-gitops doctor --cluster-config-file ./clusters --cluster prod-eu -o json

2. Read the results

doctor runs every check, even after one fails. The human output starts with the cluster name, lists each check on its own line, marked PASS, FAIL or SKIP with a short message, and ends with Doctor: healthy or the number of failed checks. The checks run in this order:

Check	What it does	When it fails
config	Confirms that the cluster config parsed. Its message summarizes the bootstrap servers, TLS on or off, the authentication mechanism and the Schema Registry endpoint, never a secret.	It always passes when doctor gets this far; a config that does not parse stops doctor with exit code 2 before any check.
kafka-connect	Builds the Kafka client: parses the connection settings and resolves the credentials. It makes no network call.	A setting is invalid or a referenced environment variable or file cannot be read.
kafka-admin	Lists the topics, the first real call to the brokers. It proves that the brokers are reachable through the Admin API and that TLS and authentication work. The message gives the number of topics.	A wrong address, or a TLS or authentication problem. If kafka-connect failed, this check is not run and fails too.
acl-read	Lists the ACLs. The message gives the number of ACLs.	The principal may not read ACLs, which takes Describe on the Cluster resource.
mds	Authenticates to Confluent MDS and lists the role bindings in the Kafka cluster’s scope. Skipped when the cluster config has no authorization.mds.	A wrong endpoint, credentials or TLS settings for MDS. An empty list of bindings is a pass.
schema-registry	Lists the Schema Registry subjects. Skipped when the cluster config has no schemaRegistry.	A wrong endpoint, credentials or TLS settings for the registry.

A skipped check does not make the result unhealthy. doctor exits 0 when no check fails, and 2 when any check fails or the command cannot start, for example because no cluster config was given, the --cluster name is unknown, or several clusters are loaded without --cluster.

In the json and yaml output, the document has the kind DoctorOutput, the cluster name, a checks list with the name, status and message of each check, and healthy, which is true when no check failed.

3. Fix what failed

Fix the first failed check first, because later checks often fail for the same reason. A kafka-connect failure points at the valueFrom references in the cluster config. A kafka-admin failure with a passing kafka-connect points at the network, TLS or credentials. Least-privilege connecting principal lists the permissions each kind of resource needs.

A healthy result does not prove that the principal may create topics, change ACLs or manage quotas and users; those permissions show up at the first apply.

The operator checks each KafkaCluster resource in a similar way and reports the result as conditions. Run this in the namespace of the KafkaCluster:

kubectl get kafkacluster prod -o jsonpath='{.status.conditions}'

Related

• The doctor command reference lists its flags.
• Validate and preview changes in CI is the next step after a healthy run.
• Runnable version: scenarios/24-doctor-preflight