--- title: "Import an existing Kafka cluster" description: "Generate monedula-gitops manifests from a live cluster with import cluster, including schemas, and verify the round trip reports no drift." url: https://monedula.dev/flock/docs/gitops/how-to/import-an-existing-cluster/ --- `import cluster` reads the live state of a Kafka cluster and writes manifests that describe it; it never changes Kafka. Running `verify` against the same cluster afterwards reports no drift. ## Before you start - The CLI with a cluster config file, `clusters/prod.yaml` on this page. - A connecting principal that may read what you import: `Describe` and `DescribeConfigs` on the topics, `Describe` on the `Cluster` resource for ACLs and SCRAM credentials, and `DescribeConfigs` on the `Cluster` resource for quotas. A super-user reads everything. - To import schemas, a `schemaRegistry` block in the cluster config. To import Confluent role bindings, an `authorization.mds` block and `rbac` in `authorization.accessBackends` ([Manage Confluent RBAC](/flock/docs/gitops/how-to/manage-confluent-rbac/#compile-topic-access-to-role-bindings)); without both, role bindings are not imported. - On Confluent Cloud, add `--skip-quotas` and `--skip-users` ([Supported platforms](/flock/docs/gitops/concepts/supported-platforms/)). ## Steps ### 1. Import into a directory ```bash monedula-gitops import cluster --cluster-config-file ./clusters/prod.yaml --output-dir ./imported ``` The import writes one file per resource under `//`, for example `imported/default/topics/payments.orders.yaml`. The kind directories are `topics`, `access`, `quotas`, `rolebindings`, `users` and `schemas`. Every manifest refers to the cluster by the `metadata.name` of the `KafkaCluster` in your cluster config, and the import prints a summary with the number of resources of each kind and any warnings. It generates: - A `KafkaTopic` per topic, with its partitions and configuration. Kafka and Confluent housekeeping topics, such as `__consumer_offsets`, `_schemas` and `_confluent*`, are skipped. - Producer and consumer ACLs that match a topic exactly become that topic's `access` block. Prefixed or ambiguous ACLs become rules of a `KafkaAccessPolicy`, and the summary warns about them. - A `KafkaQuota` per client quota entity. - A `KafkaUser` per SCRAM username, except the connecting principal's own credential. - Schema subjects named `-value` and `-key`, folded into the topic's `spec.schema`, with each schema body written verbatim under `schemas/`. A subject named `-`, where the record name matches its schema body, becomes the topic's value schema with `subjectStrategy: TopicRecordName`. If two record-based subjects map to one topic, the import keeps one and the summary warns about the other. - The live role bindings, when imported. A `DeveloperWrite` on a topic becomes a producer in that topic's `access` block, and a consumer folds in only when the principal has exactly one topic `DeveloperRead` and one group `DeveloperRead`. Everything else, such as `ResourceOwner`, cluster-scoped roles, Schema Registry, Connect and ksqlDB scopes, custom roles and prefixed patterns, becomes a `KafkaRoleBinding`. If the folded `access` blocks would not reproduce the live ACLs and role bindings exactly, the import writes every ACL as a `KafkaAccessPolicy` rule and every role binding as a `KafkaRoleBinding` instead, and the summary warns. Without `--output-dir`, the import writes the manifests to stdout and the summary to stderr, so you can pipe the manifests. It refuses stdout when the cluster has schema subjects to import, because schema bodies must be written as files. ### 2. Choose what to import `--skip-schemas`, `--skip-users` and `--skip-quotas` leave out a kind that something else owns, such as schemas that producer pipelines register, or that the cluster cannot serve, such as quotas on Confluent Cloud; `--skip-schemas` also avoids the Schema Registry connection. `--include-internal` adds the housekeeping topics. `--include-connecting-user` adds the connecting principal's own SCRAM credential, which is skipped by default because managing it in a manifest risks rotating the credential monedula-gitops itself connects with. To import everything except users: ```bash monedula-gitops import cluster --cluster-config-file ./clusters/prod.yaml --output-dir ./imported --skip-users ``` ### 3. Choose namespaces and overwrite behavior `--namespace-strategy` decides the namespace of each imported topic, and so the directory it is written to: `single` (default) uses `--namespace` (default `default`), `prefix` uses the first segment of the topic name before `--prefix-separator` (default `.`), `regex` uses the one capture group of `--namespace-regex`, and `mapping-file` reads a YAML or JSON map from topic to namespace from `--namespace-mapping-file`. An access policy follows the namespace of the topics its rules name. When you import again into an existing directory, `--overwrite` decides what happens to files that already exist: `never`, the default, skips them, `changed` rewrites a file only when its generated content differs, and `always` rewrites every file. ### 4. Finish the manifests Review the summary and the generated files before you commit them: - Each imported `KafkaUser` reads its password from a placeholder environment variable named `KAFKA_USER__PASSWORD`, because Kafka never returns SCRAM passwords; the summary warns that they are not recoverable. Set the variables, or switch the manifests to another password source, before you apply them. A user with credentials under both SCRAM mechanisms is imported with `SCRAM-SHA-512`, and the summary warns about the `SCRAM-SHA-256` one, which `verify` and `apply` then leave alone. - Subjects named after the record alone, the `RecordName` strategy, cannot be tied to a topic from cluster state. The summary lists them for manual attribution, and they are never written into a manifest. Other subjects that match no topic appear in a warning. - Check the `KafkaAccessPolicy` rules that the summary warns about. They hold the ACLs that could not be folded into a topic's `access` block, such as prefixed patterns. - The manifests are written for the CLI: topics read schema bodies with `valueFrom.file`, and users read passwords with `valueFrom.env`. The operator resolves neither, so switch schema bodies to `inline`, `configMapKeyRef` or `secretKeyRef`, and passwords to `secretKeyRef` or `generate`, before you apply them as custom resources. ## Check it worked The imported files are nested in subdirectories, so verify them with `-R`: ```bash monedula-gitops verify -f ./imported -R --cluster-config-file ./clusters/prod.yaml # exits 0: no drift ``` `verify` exits 0: the manifests describe the live topics, ACLs, quotas, users, role bindings and schemas, including each schema's subject, body and compatibility level. Passwords are never part of drift, so the placeholders do not affect the result. ## Related - The [import cluster command reference](/flock/docs/gitops/reference/cli/import-cluster/) lists every flag. - [Detect and fix drift](/flock/docs/gitops/how-to/detect-and-fix-drift/) keeps the imported state in line from then on. - Runnable versions: [scenarios/16-import-round-trip](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/16-import-round-trip) and [scenarios/23-schema-import-round-trip](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/23-schema-import-round-trip)