Monedula
monedula-gitops docs menu

Detect and fix drift

Schedule verify to catch out-of-band Kafka changes, re-apply to converge, ignore fields you do not own, and report drift without failing.

How-to guides 11 of 18

3 min read

Updated 5 October 2026

Markdown

On this page

With the CLI, a scheduled verify finds drift and apply sets the cluster back; the operator does both on its own. Drift and reconciliation modes defines what counts as drift.

Steps

1. Run verify on a schedule

This topic declares cleanup.policy: compact. Save it as manifests/topic.yaml and apply it:

apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: drift-demo
spec:
  clusterRef:
    name: shared
  topicName: drift.demo
  partitions: 1
  config:
    cleanup.policy: compact

Then run verify on a schedule, for example in a nightly CI job, with the same manifests and cluster config that your pipeline applies:

monedula-gitops verify -f ./manifests --cluster-config-file ./cluster.yaml

verify prints every difference it finds. If someone sets cleanup.policy on drift.demo to delete on the broker, it exits 1 and names cleanup.policy in its output; add -o json or -o yaml for a machine-readable report. An in-scope ACL or role binding that no manifest declares any more is drift too: verify reports it as a PruneDisabled candidate until you prune it or declare it again.

2. Converge with apply

Preview the operations that apply would run:

monedula-gitops diff -f ./manifests --cluster-config-file ./cluster.yaml

Then apply:

monedula-gitops apply -f ./manifests --cluster-config-file ./cluster.yaml

After the apply, verify exits 0 again. Three kinds of difference do not converge with a plain apply:

  • A change that needs approval is reported as Blocked, and apply exits 3. For example, a limit someone added by hand to a quota entity your manifests declare is removed only with --allow-destructive.
  • A partition increase made outside monedula-gitops cannot be reverted: apply rejects the decrease and exits 2. Set partitions in the manifest to the live count, or ignore the field.
  • A prune candidate stays until you apply with --prune.

Ignore fields you do not own

When another system is meant to change a setting, exclude it with spec.drift.ignoreFields. Here retention.ms is set on creation and left to its other owner afterwards:

apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: ignore-demo
spec:
  clusterRef:
    name: shared
  topicName: ignore.demo
  partitions: 1
  config:
    cleanup.policy: compact
    retention.ms: "604800000"
  drift:
    ignoreFields:
      - config.retention.ms

Valid entries are partitions, replicationFactor and config.<key>; anything else fails validation. If retention.ms on ignore.demo changes on the broker, verify still exits 0 and apply leaves the live value alone. A change to cleanup.policy is still drift.

Report drift without failing

To audit a resource without enforcing it, set its reconciliation mode to ObserveOnly:

apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: observe-demo
spec:
  clusterRef:
    name: shared
  topicName: observe.demo
  partitions: 1
  config:
    cleanup.policy: compact
  reconciliation:
    mode: ObserveOnly

verify shows the drift but exits 0, and apply reports the operations as ReportOnly without running them. It never creates anything either, so create the topic first with the same manifest without reconciliation. KafkaAccessPolicy, KafkaQuota and KafkaRoleBinding take the same field.

To fail a check on drift without correcting it, use DetectOnly instead. For a report that never fails a job on drift, run diff, which exits 0 with or without drift and 2 on error.

Let the operator correct drift

The operator compares each resource with the cluster on every reconcile and on every resync. In the default Enforce mode it reverts an out-of-band change at the next resync. Changes that need approval wait for the matching annotation, and prune candidates for spec.prune: true. spec.drift.ignoreFields works the same way; the modes table shows what a DetectOnly or ObserveOnly resource reports.

Check it worked

monedula-gitops verify -f ./manifests --cluster-config-file ./cluster.yaml
kubectl get kafkatopic drift-demo -o jsonpath='{.status.conditions}'

verify exits 0, and drift.demo has cleanup.policy: compact again. With the operator, the Ready condition has the status True and DriftDetected the status False.