--- title: "Recover from a failed run" description: "What to do when apply stops halfway, a lock is left behind, or verify reports missing access: rollback scripts, unlocks and re-runs." url: https://monedula.dev/flock/docs/acl-rbac-converter/how-to/recover-from-a-failed-run/ --- Every step of a migration leaves its record in the run directory, so a run that stopped halfway can be finished or undone from there. Find where it stopped, then follow the section that matches: most failures end with running the same command again. ## Find where the run stopped ```bash monedula-acl-rbac status runs/2026-05-21T10-00-00Z ``` `status` shows which steps left artifacts, whether `plan.json` still matches its checksum, the `verify` counts and any lock ([Check on a previous run](/flock/docs/acl-rbac-converter/how-to/check-a-previous-run/) explains its output). Then read the log of the step that failed: - `apply.log`: every MDS call of `apply` and its result, as `CREATE`, `SKIP` or `FAIL` lines. - `verify.json`: the result of every source ACL. - `delete.log` and `delete-deny.log`: one `OK`, `SKIP` or `FAIL` line per ACL from the deletion scripts. ## Apply stopped halfway Run `apply` again with the same plan: ```bash monedula-acl-rbac apply \ --plan runs/2026-05-21T10-00-00Z/plan.json \ --mds-url https://mds.example.com \ --mds-token-file ~/.confluent/token \ --confirm ``` `apply` skips the bindings that already exist and creates the rest. The `FAIL` lines in `apply.log` tell you what went wrong. Raise `--mds-max-retries` (default 3) on a flaky connection; an authentication error means the token or password needs renewing. ## A lock is left behind `apply` holds a lock file, `.lock`, in the run directory while it runs. If a run was killed, the lock stays. While a lock exists, the next `apply` refuses to start, with exit code 4, and the message tells you which case you are in: - The lock belongs to a process that is still running on this host: another `apply` is in progress. Wait for it. - The lock is stale, because its process is gone or it was taken on another host: `apply` names the stale lock and asks for `--force-unlock`. `status` shows the lock's contents: the process ID, host name, command and start time. Once you are sure that process is gone, add `--force-unlock` to the `apply` command. `--force-unlock` removes the lock even when its process looks alive. Clearing the lock of a running `apply` lets two runs write to MDS from the same plan at once. ## Verify reports missing access `verify` exits 5 when any result is `EFFECTIVE_MISSING` or `EFFECTIVE_UNKNOWN`, and `delete-acls` leaves those ACLs out. List them: ```bash jq '.results[] | select(.status != "EFFECTIVE_OK")' verify.json ``` [Verify effective access](/flock/docs/acl-rbac-converter/how-to/verify-effective-access/) lists the usual causes and their fixes, and the `--accept-unknown-verify` option for an MDS that cannot answer effective-access lookups. When the fix changes the plan, as a new principal mapping or cluster ID does, run `plan`, `apply` and `verify` again: the delete commands refuse a `verify.json` made for an older plan. ## A deletion script stopped Both deletion scripts stop at the first failure. Read `delete.log` or `delete-deny.log`: the ACLs with an `OK` line are deleted, and the `FAIL` line shows where the script stopped. - If the script refused to start with exit code 5, its [`plan.json` hash check](/flock/docs/acl-rbac-converter/concepts/safety-guarantees/#stale-plans-are-refused) failed or, for the DENY script, `runtime.env` was missing or not mode 0600. Generate the script again with `delete-acls` or `delete-deny-acls`. - For `delete-acls.sh`, fix the cause of the failure, then run the lines that did not log `OK`. The script is plain bash, so you can edit it, for example to drop the lines that already succeeded. - `delete-deny-acls.sh` deletes its `runtime.env` when it exits, so it cannot run a second time. Generate it again with `delete-deny-acls`; each DENY is checked against MDS again when the new script runs. To undo deletions, run the rollback script of the same run directory against the same cluster. `rollback.sh` recreates the ACLs that `delete-acls.sh` would remove or did remove, and `rollback-deny.sh` the DENY ACLs: ```bash bash runs/2026-05-21T10-00-00Z/rollback.sh ``` ## Roll back role bindings The tool does not delete role bindings. To remove bindings that `apply` created, use the Confluent CLI or the MDS API directly. `apply.log` lists every binding it created, and `plan.json` holds each binding's principal, role, scope and resources. For bindings applied by a pipeline from emitted artifacts, see the rollback step of [Hand bindings to a CD pipeline](/flock/docs/acl-rbac-converter/how-to/emit-for-a-cd-pipeline/). ## Check it worked `status` shows `lock: none`, `checksum_ok=true` on the `plan` line and `0 missing, 0 unknown` on the `verify` line. The step you repeated exits 0, and its log has no new `FAIL` line. ## Related - The [CLI overview](/flock/docs/acl-rbac-converter/reference/cli/) lists the exit codes and the files of a run directory.