---
title: "Detect and fix drift"
description: "Schedule verify to catch out-of-band Kafka changes, re-apply to converge, ignore fields you do not own, and report drift without failing."
url: https://monedula.dev/flock/docs/gitops/how-to/detect-and-fix-drift/
---
With the CLI, a scheduled `verify` finds drift and `apply` sets the cluster back; the operator does both on its own. [Drift and reconciliation modes](/flock/docs/gitops/concepts/drift-and-reconciliation/) defines what counts as drift.

## Steps

### 1. Run verify on a schedule

This topic declares `cleanup.policy: compact`. Save it as `manifests/topic.yaml` and apply it:

```yaml
apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: drift-demo
spec:
  clusterRef:
    name: shared
  topicName: drift.demo
  partitions: 1
  config:
    cleanup.policy: compact
```

Then run `verify` on a schedule, for example in a nightly CI job, with the same manifests and cluster config that your pipeline applies:

```bash
monedula-gitops verify -f ./manifests --cluster-config-file ./cluster.yaml
```

`verify` prints every difference it finds. If someone sets `cleanup.policy` on `drift.demo` to `delete` on the broker, it exits 1 and names `cleanup.policy` in its output; add `-o json` or `-o yaml` for a machine-readable report. An in-scope ACL or role binding that no manifest declares any more is drift too: `verify` reports it as a `PruneDisabled` candidate until you prune it or declare it again.

### 2. Converge with apply

Preview the operations that `apply` would run:

```bash
monedula-gitops diff -f ./manifests --cluster-config-file ./cluster.yaml
```

<Callout type="warning">
Re-applying reverts every out-of-band change that drift detection sees, whoever made it and for whatever reason. If the change was intended, put it in the manifest instead, or ignore the field as described below.
</Callout>

Then apply:

```bash
monedula-gitops apply -f ./manifests --cluster-config-file ./cluster.yaml
```

After the apply, `verify` exits 0 again. Three kinds of difference do not converge with a plain `apply`:

- A change that needs approval is reported as `Blocked`, and `apply` exits 3. For example, a limit someone added by hand to a quota entity your manifests declare is removed only with `--allow-destructive`.
- A partition increase made outside monedula-gitops cannot be reverted: `apply` rejects the decrease and exits 2. Set `partitions` in the manifest to the live count, or ignore the field.
- A prune candidate stays until you apply with `--prune`.

## Ignore fields you do not own

When another system is meant to change a setting, exclude it with `spec.drift.ignoreFields`. Here `retention.ms` is set on creation and left to its other owner afterwards:

```yaml
apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: ignore-demo
spec:
  clusterRef:
    name: shared
  topicName: ignore.demo
  partitions: 1
  config:
    cleanup.policy: compact
    retention.ms: "604800000"
  drift:
    ignoreFields:
      - config.retention.ms
```

Valid entries are `partitions`, `replicationFactor` and `config.<key>`; anything else fails validation. If `retention.ms` on `ignore.demo` changes on the broker, `verify` still exits 0 and `apply` leaves the live value alone. A change to `cleanup.policy` is still drift.

## Report drift without failing

To audit a resource without enforcing it, set its reconciliation mode to `ObserveOnly`:

```yaml
apiVersion: gitops.monedula.dev/v1alpha1
kind: KafkaTopic
metadata:
  name: observe-demo
spec:
  clusterRef:
    name: shared
  topicName: observe.demo
  partitions: 1
  config:
    cleanup.policy: compact
  reconciliation:
    mode: ObserveOnly
```

`verify` shows the drift but exits 0, and `apply` reports the operations as `ReportOnly` without running them. It never creates anything either, so create the topic first with the same manifest without `reconciliation`. `KafkaAccessPolicy`, `KafkaQuota` and `KafkaRoleBinding` take the same field.

To fail a check on drift without correcting it, use `DetectOnly` instead. For a report that never fails a job on drift, run `diff`, which exits 0 with or without drift and 2 on error.

## Let the operator correct drift

The operator compares each resource with the cluster on every reconcile and on every [resync](/flock/docs/gitops/how-to/configure-the-operator/). In the default `Enforce` mode it reverts an out-of-band change at the next resync. Changes that need approval wait for the matching annotation, and prune candidates for `spec.prune: true`. `spec.drift.ignoreFields` works the same way; the [modes table](/flock/docs/gitops/concepts/drift-and-reconciliation/) shows what a `DetectOnly` or `ObserveOnly` resource reports.

## Check it worked

<Tabs group="detect-and-fix-drift-check" labels={['CLI', 'Operator']} label="Where you run it">

```bash
monedula-gitops verify -f ./manifests --cluster-config-file ./cluster.yaml
```

```bash
kubectl get kafkatopic drift-demo -o jsonpath='{.status.conditions}'
```

</Tabs>

`verify` exits 0, and `drift.demo` has `cleanup.policy: compact` again. With the operator, the `Ready` condition has the status `True` and `DriftDetected` the status `False`.

## Related

- The [verify command reference](/flock/docs/gitops/reference/cli/verify/) lists its flags, and the [KafkaTopic reference](/flock/docs/gitops/reference/resources/kafka-topic/) the `drift` and `reconciliation` fields.
- Runnable versions: [scenarios/10-drift-detect-reconcile](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/10-drift-detect-reconcile), [scenarios/11-reconciliation-modes](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/11-reconciliation-modes) and [scenarios/19-drift-ignore-fields](https://github.com/monedula-dev/monedula-gitops/tree/main/scenarios/19-drift-ignore-fields)