Recover from a failed run

Every step of a migration leaves its record in the run directory, so a run that stopped halfway can be finished or undone from there. Find where it stopped, then follow the section that matches: most failures end with running the same command again.

Find where the run stopped

monedula-acl-rbac status runs/2026-05-21T10-00-00Z

status shows which steps left artifacts, whether plan.json still matches its checksum, the verify counts and any lock (Check on a previous run explains its output). Then read the log of the step that failed:

• apply.log: every MDS call of apply and its result, as CREATE, SKIP or FAIL lines.
• verify.json: the result of every source ACL.
• delete.log and delete-deny.log: one OK, SKIP or FAIL line per ACL from the deletion scripts.

Apply stopped halfway

Run apply again with the same plan:

monedula-acl-rbac apply \
  --plan runs/2026-05-21T10-00-00Z/plan.json \
  --mds-url https://mds.example.com \
  --mds-token-file ~/.confluent/token \
  --confirm

apply skips the bindings that already exist and creates the rest. The FAIL lines in apply.log tell you what went wrong. Raise --mds-max-retries (default 3) on a flaky connection; an authentication error means the token or password needs renewing.

A lock is left behind

apply holds a lock file, .lock, in the run directory while it runs. If a run was killed, the lock stays. While a lock exists, the next apply refuses to start, with exit code 4, and the message tells you which case you are in:

• The lock belongs to a process that is still running on this host: another apply is in progress. Wait for it.
• The lock is stale, because its process is gone or it was taken on another host: apply names the stale lock and asks for --force-unlock.

status shows the lock’s contents: the process ID, host name, command and start time. Once you are sure that process is gone, add --force-unlock to the apply command.

Verify reports missing access

verify exits 5 when any result is EFFECTIVE_MISSING or EFFECTIVE_UNKNOWN, and delete-acls leaves those ACLs out. List them:

jq '.results[] | select(.status != "EFFECTIVE_OK")' verify.json

Verify effective access lists the usual causes and their fixes, and the --accept-unknown-verify option for an MDS that cannot answer effective-access lookups. When the fix changes the plan, as a new principal mapping or cluster ID does, run plan, apply and verify again: the delete commands refuse a verify.json made for an older plan.

A deletion script stopped

Both deletion scripts stop at the first failure. Read delete.log or delete-deny.log: the ACLs with an OK line are deleted, and the FAIL line shows where the script stopped.

• If the script refused to start with exit code 5, its plan.json hash check failed or, for the DENY script, runtime.env was missing or not mode 0600. Generate the script again with delete-acls or delete-deny-acls.
• For delete-acls.sh, fix the cause of the failure, then run the lines that did not log OK. The script is plain bash, so you can edit it, for example to drop the lines that already succeeded.
• delete-deny-acls.sh deletes its runtime.env when it exits, so it cannot run a second time. Generate it again with delete-deny-acls; each DENY is checked against MDS again when the new script runs.

To undo deletions, run the rollback script of the same run directory against the same cluster. rollback.sh recreates the ACLs that delete-acls.sh would remove or did remove, and rollback-deny.sh the DENY ACLs:

bash runs/2026-05-21T10-00-00Z/rollback.sh

Roll back role bindings

The tool does not delete role bindings. To remove bindings that apply created, use the Confluent CLI or the MDS API directly. apply.log lists every binding it created, and plan.json holds each binding’s principal, role, scope and resources. For bindings applied by a pipeline from emitted artifacts, see the rollback step of Hand bindings to a CD pipeline.

Check it worked

status shows lock: none, checksum_ok=true on the plan line and 0 missing, 0 unknown on the verify line. The step you repeated exits 0, and its log has no new FAIL line.

Related

• The CLI overview lists the exit codes and the files of a run directory.