--- title: "Register and evolve schemas" description: "Register Schema Registry subjects from a KafkaTopic, set compatibility, and evolve schemas safely; the registry rejects incompatible changes." url: https://monedula.dev/flock/docs/gitops/how-to/manage-schemas/ --- Declare a topic's schema in its `KafkaTopic`, and monedula-gitops registers it in Schema Registry and sets the subject's compatibility level, in the same diff, verify and apply flow as the topic itself. The compatibility check on a new version is left to the registry. ## Before you start - A `spec.schemaRegistry` connection on the `KafkaCluster`, as in [Connect to a secured Kafka cluster](/flock/docs/gitops/how-to/connect-to-a-secured-cluster/), to any registry that speaks the Confluent Schema Registry REST API. Without it, a topic with a `schema` block fails validation: the CLI exits 2 with `spec.schema requires KafkaCluster "shared" to configure schemaRegistry`, and the operator sets `ValidationFailed=True` on the topic. - A registry identity that may read and register subjects, and read and write their compatibility configuration. ## Steps ### 1. Declare the schema Add a `schema` block to the topic. This manifest creates the topic `schema.evo` and registers an Avro record schema, given inline, with `BACKWARD` compatibility. Save it as `manifests/topic.yaml`: ```yaml apiVersion: gitops.monedula.dev/v1alpha1 kind: KafkaTopic metadata: name: schema-evo spec: clusterRef: name: shared topicName: schema.evo partitions: 1 schema: format: AVRO compatibility: BACKWARD valueSchema: valueFrom: inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"long"}]}' ``` - `format` is `AVRO`, `JSON` or `PROTOBUF`. - `compatibility` is the subject-level compatibility: `NONE`, `BACKWARD`, `BACKWARD_TRANSITIVE`, `FORWARD`, `FORWARD_TRANSITIVE`, `FULL` or `FULL_TRANSITIVE`. - `valueSchema` holds the value schema body, and the optional `keySchema` the key schema body. monedula-gitops sends the body to the registry verbatim. - `subjectStrategy` decides the subject names. The default, `TopicName`, uses `-value` and `-key`, so this schema goes to the subject `schema.evo-value`. `RecordName` uses the record's full name from the schema body (the Avro `namespace` and `name`, the JSON Schema `title`, or the Protobuf `package` and first `message`), and `TopicRecordName` uses `-`; both need a `valueSchema`. `Custom` takes the names verbatim from `valueSubject`, which it requires, and `keySubject`, which it requires when a `keySchema` is set. Any other strategy rejects `valueSubject` and `keySubject`. `valueFrom.inline`, as above, works in both runtimes. The CLI also reads `valueFrom.file`, resolved relative to the manifest, such as `file: ./schemas/orders-value.avsc`. The operator cannot resolve `valueFrom.file` or `valueFrom.env`: it sets `SchemaSynced=False` with the reason `SchemaUnresolved` and still reconciles the topic and its access. With the operator, use `inline` or `valueFrom.configMapKeyRef` with a ConfigMap in the topic's namespace, and label the ConfigMap `gitops.monedula.dev/schema-source: "true"` so the operator picks up edits at once. Without the label, edits land at the next [resync](/flock/docs/gitops/how-to/configure-the-operator/) (default `5m`), and the topic reports `SchemaSourceUnwatched=True` with the reason `ConfigMapNotLabeled`, which does not block the reconcile. To govern only the compatibility level, leave out `valueSchema` and `keySchema` and keep `format` and `compatibility`, which this mode requires. monedula-gitops then never registers a version, and the versions that producer pipelines register are not drift. This governance mode is the recommended default; supply a `valueSchema` only for contract-first topics. ### 2. Apply the manifest ```bash monedula-gitops apply -f ./manifests --cluster-config-file ./cluster.yaml ``` ```bash kubectl apply -f manifests/ ``` monedula-gitops creates the topic, registers the schema as the first version of `schema.evo-value` and sets the subject's compatibility to `BACKWARD`. ### 3. Evolve the schema Change the schema in the manifest and apply again. This version adds a `label` field with a default value, a `BACKWARD`-compatible change: ```yaml spec: schema: valueSchema: valueFrom: inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"long"},{"name":"label","type":"string","default":""}]}' ``` The registry accepts it as version 2. Registering a new version needs no approval. If the new version retypes `id` from `long` to `string` instead, it is incompatible: ```yaml inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"string"}]}' ``` The registry rejects this version as incompatible with `BACKWARD`. `apply` exits 2, and its output contains `compat`. With the operator, the topic's `SchemaSynced` condition turns `False`. ### 4. Change the compatibility level Raising the level, for example from `BACKWARD` to `FULL`, applies directly. Lowering it or moving sideways, for example from `FULL` to `BACKWARD` or from `BACKWARD` to `FORWARD`, is [gated](/flock/docs/gitops/concepts/safety-gates/): `apply` reports it as `Blocked` and exits 3 until you approve it with `--allow-destructive`, or with the annotation `gitops.monedula.dev/allow-destructive: "true"` when you use the operator. The first level you declare on a subject counts as a lowering when it is below the registry's global default, because a subject without its own level runs at that default. For example, `NONE` on a new subject under a global `BACKWARD` is gated. Declare a level at or above the global default to avoid the approval. monedula-gitops reads the global default (`GET /config`) once per CLI run and once per operator reconcile. If that read fails, every first-time level counts as a raise and applies without approval: the CLI logs a warning and continues, and the operator sets `SchemaRegistryDegraded=True` with the reason `GlobalCompatibilityFetchFailed` on the topic. Removing the `schema` block from a manifest leaves the subject in the registry. ## Check it worked ```bash monedula-gitops verify -f ./manifests --cluster-config-file ./cluster.yaml ``` ```bash kubectl get kafkatopic schema-evo -o jsonpath='{.status.conditions}' ``` `verify` exits 0 when the topic and its subject match the manifest. With the operator, the `Ready` and `SchemaSynced` conditions have the status `True`. The subject `schema.evo-value` is registered with `BACKWARD` compatibility. ## Related - The [KafkaTopic reference](/flock/docs/gitops/reference/resources/kafka-topic/) lists every `schema` field. - [Import an existing Kafka cluster](/flock/docs/gitops/how-to/import-an-existing-cluster/) brings existing subjects under management. - Runnable versions: [scenarios/09-register-schema](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/09-register-schema) runs in both runtimes, and [scenarios/18-schema-evolution](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/18-schema-evolution) walks through the evolution steps.