Register and evolve schemas

Register Schema Registry subjects from a KafkaTopic, set compatibility, and evolve schemas safely; the registry rejects incompatible changes.

Declare a topic’s schema in its KafkaTopic, and monedula-gitops registers it in Schema Registry and sets the subject’s compatibility level, in the same diff, verify and apply flow as the topic itself. The compatibility check on a new version is left to the registry.

Before you start

A spec.schemaRegistry connection on the KafkaCluster, as in Connect to a secured Kafka cluster, to any registry that speaks the Confluent Schema Registry REST API. Without it, a topic with a schema block fails validation: the CLI exits 2 with spec.schema requires KafkaCluster "shared" to configure schemaRegistry, and the operator sets ValidationFailed=True on the topic.
A registry identity that may read and register subjects, and read and write their compatibility configuration.

Steps

1. Declare the schema

Add a schema block to the topic. This manifest creates the topic schema.evo and registers an Avro record schema, given inline, with BACKWARD compatibility. Save it as manifests/topic.yaml:

apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: schema-evo
spec:
  clusterRef:
    name: shared
  topicName: schema.evo
  partitions: 1
  schema:
    format: AVRO
    compatibility: BACKWARD
    valueSchema:
      valueFrom:
        inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"long"}]}'

format is AVRO, JSON or PROTOBUF.
compatibility is the subject-level compatibility: NONE, BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL or FULL_TRANSITIVE.
valueSchema holds the value schema body, and the optional keySchema the key schema body. monedula-gitops sends the body to the registry verbatim.
subjectStrategy decides the subject names. The default, TopicName, uses <topicName>-value and <topicName>-key, so this schema goes to the subject schema.evo-value. RecordName uses the record’s full name from the schema body (the Avro namespace and name, the JSON Schema title, or the Protobuf package and first message), and TopicRecordName uses <topicName>-<recordFullName>; both need a valueSchema. Custom takes the names verbatim from valueSubject, which it requires, and keySubject, which it requires when a keySchema is set. Any other strategy rejects valueSubject and keySubject.

valueFrom.inline, as above, works in both runtimes. The CLI also reads valueFrom.file, resolved relative to the manifest, such as file: ./schemas/orders-value.avsc. The operator cannot resolve valueFrom.file or valueFrom.env: it sets SchemaSynced=False with the reason SchemaUnresolved and still reconciles the topic and its access. With the operator, use inline or valueFrom.configMapKeyRef with a ConfigMap in the topic’s namespace, and label the ConfigMap gitops.monedula.dev/schema-source: "true" so the operator picks up edits at once. Without the label, edits land at the next resync (default 5m), and the topic reports SchemaSourceUnwatched=True with the reason ConfigMapNotLabeled, which does not block the reconcile.

To govern only the compatibility level, leave out valueSchema and keySchema and keep format and compatibility, which this mode requires. monedula-gitops then never registers a version, and the versions that producer pipelines register are not drift. This governance mode is the recommended default; supply a valueSchema only for contract-first topics.

2. Apply the manifest

CLIOperator

monedula-gitops apply -f ./manifests --cluster-config-file ./cluster.yaml

kubectl apply -f manifests/

monedula-gitops creates the topic, registers the schema as the first version of schema.evo-value and sets the subject’s compatibility to BACKWARD.

3. Evolve the schema

Change the schema in the manifest and apply again. This version adds a label field with a default value, a BACKWARD-compatible change:

spec:
  schema:
    valueSchema:
      valueFrom:
        inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"long"},{"name":"label","type":"string","default":""}]}'

The registry accepts it as version 2. Registering a new version needs no approval.

If the new version retypes id from long to string instead, it is incompatible:

inline: '{"type":"record","name":"SchemaEvo","fields":[{"name":"id","type":"string"}]}'

The registry rejects this version as incompatible with BACKWARD. apply exits 2, and its output contains compat. With the operator, the topic’s SchemaSynced condition turns False.

4. Change the compatibility level

Raising the level, for example from BACKWARD to FULL, applies directly. Lowering it or moving sideways, for example from FULL to BACKWARD or from BACKWARD to FORWARD, is gated: apply reports it as Blocked and exits 3 until you approve it with --allow-destructive, or with the annotation gitops.monedula.dev/allow-destructive: "true" when you use the operator.

The first level you declare on a subject counts as a lowering when it is below the registry’s global default, because a subject without its own level runs at that default. For example, NONE on a new subject under a global BACKWARD is gated. Declare a level at or above the global default to avoid the approval.

monedula-gitops reads the global default (GET /config) once per CLI run and once per operator reconcile. If that read fails, every first-time level counts as a raise and applies without approval: the CLI logs a warning and continues, and the operator sets SchemaRegistryDegraded=True with the reason GlobalCompatibilityFetchFailed on the topic.

Removing the schema block from a manifest leaves the subject in the registry.

Check it worked