← braid

Recovery scenarios

Detailed walkthroughs for recovering from failures. Read this when braid status or another command tells you something is wrong, or when you are planning for failure ahead of time.

Overview: discover vs recover

braid has two recovery commands that solve different problems:

discover solves metadata loss – the CLI’s record of which disks belong to the pool is gone, but the disks themselves are fine. It reads LUKS labels (braid-<name>) and LUKS UUIDs from /dev/disk/by-id/ devices to reconstruct UUID-keyed membership.

recover solves interrupted operations – an add, remove, remove-missing, or replace was killed mid-flight (power loss, crash, OOM). The pending-operation journal (/var/lib/braid/pending-op.json) records what was in progress. Recover opens the pool, inspects what actually happened on disk, and rebuilds pool.json to match reality.

Lost pool.json

Symptom: braid unlock fails because /var/lib/braid/pool.json does not exist.

Cause: Accidental deletion, filesystem corruption, or migrating to a new NixOS install.

Steps

1. Verify no pending operation exists:

ls /var/lib/braid/pending-op.json
# If this file exists, use `braid recover` instead (see below)

2. Scan for braid disks:

sudo braid discover

Output looks like:

  toshiba1 = /dev/disk/by-id/ata-TOSHIBA_MN08ACA16T_XXXX
  toshiba2 = /dev/disk/by-id/ata-TOSHIBA_MN08ACA16T_YYYY
  toshiba3 = /dev/disk/by-id/ata-TOSHIBA_MN08ACA16T_ZZZZ

3. Verify the output matches your expected pool members. Then write:

sudo braid discover --write

This creates /var/lib/braid/pool.json.

If you can name the expected member count ahead of time, record it from your own records or prior braid status output and pass it as a fail-closed guard:

EXPECTED=3
sudo braid discover --write --expect-count="$EXPECTED"

4. Unlock normally:

sudo braid unlock

Notes

• For a healthy UUID-keyed pool.json, discover --write refuses – use braid add / braid remove / braid replace to mutate membership instead.
• For a corrupt or off-schema existing pool.json, discover --write rebuilds in place; no manual remove step is needed. The original bytes are preserved at pool.json.corrupt-<RFC3339-UTC> adjacent to the new file in case manual forensic recovery is needed (e.g. extracting a devid for a member btrfs reports missing). The snapshot is a hard precondition: if it cannot be written (full disk, read-only state directory), discover --write refuses rather than destroy the corrupt original; free disk space or fix permissions and retry.
• discover --write refuses to run if pending-op.json exists. Use braid recover instead. (Bare discover is read-only and runs regardless.)
• discover only finds LUKS2 devices. LUKS1 devices with braid labels are skipped with a warning.
• The rebuilt pool.json is keyed by LUKS UUID. Disk names are stored in each member value for command input and display.
• When multiple /dev/disk/by-id/ symlinks point to the same device, discover picks the most stable one (wwn > nvme > scsi > ata > usb).

Interrupted add/remove/replace

Symptom: braid commands fail with a message about a pending operation. ls /var/lib/braid/pending-op.json confirms the journal file exists.

Cause: A pool mutation (add, remove, remove-missing, replace) was interrupted before it could complete. The journal records the operation type, the pre-operation membership, and the target membership. Existing-pool add journals also record a phase: PoolMutation for unfinished disk preparation or btrfs membership, and PostAddBalanceRaid1 after membership is committed but balance work remains.

Steps

1. Preview what recover will do:

sudo braid recover --dry-run

This shows the recovery plan without making changes: which LUKS devices will be opened, whether the pool needs mounting, and the final pool.json state.

2. Run recovery:

sudo braid recover

Recover will:

• Open the LUKS devices needed for the journal phase
• Mount the btrfs pool
• Probe the live btrfs topology to determine what actually happened
• For existing-pool add PoolMutation, first open and scan any already-committed journaled add targets that can be reconciled without wiping or adding
• For add PoolMutation, finish only the journaled add targets that are not already live
• For add PostAddBalanceRaid1, skip all disk preparation and btrfs add steps, then run the owed RAID1 balance only when btrfs balance state is idle and the pool can host RAID1 (skip it while a member is missing or hot-unplugged); preserve pending-op.json when a paused, running, or unknown balance state requires manual inspection
• Rebuild or repair pool.json only when live membership is complete
• Clear pending-op.json only after required membership and balance work is complete, or the balance was skipped because the pool cannot host RAID1

3. Verify:

sudo braid status

If recovery itself fails, its final guidance comes from a fresh authoritative read of pending-op.json:

• A valid journal is still visible: address the reported failure, then rerun sudo braid recover. Recovery is designed to retry from the journaled phase.
• The journal is absent: there is no recovery state to retry. If recovery had just cleared the journal, deletion durability was not confirmed; repair state-directory I/O, restart, and run recovery only if pending-op.json reappears.
• The journal cannot be read or parsed: do not keep rerunning recovery. Repair state-directory access, or follow Pending-op file corruption for manual reconciliation of invalid contents.

Interrupted between returned-disk wipe and add

If an existing braid-labeled disk was being returned to the pool and the add was interrupted after wipefs --types btrfs but before btrfs device add, run:

sudo braid recover

Recover replays the add from the journaled returned-disk target. Do not wipe the disk and retry it as a fresh add; the journal still records the checked LUKS identity and expected pool FSID.

If the disk is absent, recover fails and leaves pending-op.json in place. Reconnect the original disk, then rerun sudo braid recover.

Interrupted fresh-disk add

For an interrupted fresh-disk add, recover finishes the steps braid add planned for the disk when luksFormat already wrote it. Interrupted adds lists them.

Recover never formats a disk. If the disk is still not LUKS, luksFormat never wrote it, so recover drops it from pending-op.json and you rerun braid add for it. That covers a --luks-format-arg option that made luksFormat fail after braid wrote the journal. If the dropped disk was the add’s only target, recover restores the pre-add pool.json and clears the journal.

If the disk is absent or has a different LUKS label than the journal records, recover fails and leaves pending-op.json in place. Reconnect the original disk or replace the target, then rerun sudo braid recover.

Interrupted first add (bootstrap)

The first braid add creates the pool with mkfs.btrfs after it formats every disk. If it stopped before mkfs.btrfs, there is no pool, and sudo braid recover clears pending-op.json instead of mounting. If luksFormat wrote some disks, recover names them; wipe each one (sudo wipefs -a <by-id path>) before you rerun braid add, since add refuses a braid-labeled disk with no btrfs.

When the pool is not mounted, recover checks every journaled disk before it decides whether mkfs.btrfs ran. It refuses and keeps pending-op.json if a disk is gone (recover add target '<name>' (<by-id path>) is not present; reconnect it), or if a disk carries a LUKS header without the braid-<name> label or the journaled LUKS UUID. A gone disk refuses this way even if mkfs.btrfs ran, with or without --allow-degraded, and recover exits 1. Reconnect the disk, then rerun sudo braid recover.

Pending-op file corruption

Symptom: braid reports that /var/lib/braid/pending-op.json could not be parsed.

The remediation phrase is:

Remove /var/lib/braid/pending-op.json after manual reconciliation (see docs/internals/luks-unlock.md) and re-run.

It is safe to remove pending-op.json only when one of these is true:

When it is not safe, keep the journal in place and investigate the interrupted operation before editing state.

Out-of-band reformat during recovery

Recover refuses if a target disk’s live LUKS UUID no longer matches the journal. This catches a disk that was reformatted, swapped, or cloned after the original operation started.

Messages to search for:

• add recovery aborted: target ... LUKS UUID mismatch
• recover replace target '...' LUKS UUID mismatch: expected ..., found ...

Do not force the journal forward. Investigate the foreign reformat or swapped disk, restore the intended disk if possible, and rerun recovery.

See also Unlock refused by a foreign or mismatched disk for the same identity check on the braid unlock path.

Null-underlying mapper during recovery

A member can stay in the live pool after its LUKS backing device is gone (cryptsetup status reports device: (null)). Recovery of every operation still binds that mapper to its UUID-keyed membership entry, because the kernel keeps the LUKS UUID the mapper was opened with; no persisted devid is needed. The member stays in pool.json at the devid btrfs holds it at. This holds for a disk the interrupted braid add was adding, too: if btrfs already added it, recover counts it as added and does not add it again.

If that UUID belongs to no journaled member, recovery refuses and keeps pending-op.json. For an interrupted braid remove the refusal reads device <mapper> (LUKS UUID <uuid>) is in the live pool with no backing disk but is not part of the pre-operation membership. Find out which disk that mapper belonged to before you re-run braid recover.

Once btrfs promotes a gone device to MISSING, braid remove-missing or braid replace can act on it.

Duplicate or missing devid in journal snapshot

btrfs reports a device it cannot reach as a MISSING devid with no LUKS UUID. Recovery binds that devid to the journaled member that records it. A MISSING devid that no journaled member records is left out of pool.json, as braid add leaves it.

Recovery refuses and keeps pending-op.json when it cannot bind a MISSING devid safely:

• Two journaled members record the same devid: duplicate devid <devid> in journaled membership across UUIDs ....
• A MISSING devid no member records sits beside a journaled member that records no devid, so either could be the other: recovery cannot safely correlate journaled member(s) '<name>' with btrfs-MISSING devid(s) <devid>: .... A bootstrap braid add journal records no devids, so a bootstrap whose disk is gone hits this refusal even with --allow-degraded when the pool is already mounted. When it is not mounted, recover refuses earlier, at the gone disk; see Interrupted first add (bootstrap).

Do not edit pool.json; that resolution did not consult it. Re-run recovery only after manual reconciliation of pending-op.json. If the disk is reachable, reattach it and re-run braid recover; recovery then binds it by its LUKS UUID.

Committed-but-closed add target

If the journaled add target is already a live pool member but its mapper is closed when recover starts, recover opens and scans it during the reconciliation pass. After the live-pool re-probe, the target is included in pool.json and is not re-added. If the target’s disk is gone instead, recover cannot open it and refuses; see With missing devices.

This can still prompt for the pool passphrase even when the pool is already mounted, because the target mapper may need to be opened before recover can discover that it already committed.

With missing devices

If a drive failed during the interrupted operation:

sudo braid recover --allow-degraded

Without --allow-degraded, recover exits with code 2 when devices are missing. An interrupted bootstrap braid add with the pool not mounted is the exception: recover refuses at the gone disk before it mounts, and exits 1. The degraded flag allows mounting with missing devices so recovery can complete. The missing member stays in pool.json, so braid replace or braid remove-missing can name it later. Redundancy is reduced until the missing device is replaced.

Recovery can finish a degraded pool only when it can tell which member each MISSING devid is. An interrupted bootstrap braid add (the pool’s first disks) records no devids, so --allow-degraded cannot finish it while one of its disks is gone. With the pool not mounted, recover refuses at the gone disk (Interrupted first add (bootstrap)); with it mounted, recover refuses at the MISSING devid (Duplicate or missing devid in journal snapshot).

An interrupted existing-pool braid add in its PoolMutation phase cannot finish while a disk it was adding is gone, even if btrfs already added that disk. The journal records no devid for a new disk until the advance to PostAddBalanceRaid1, so recover cannot tell that a MISSING devid is that disk. It refuses at that disk and keeps pending-op.json, with or without --allow-degraded. Reattach the disk and re-run braid recover. After the advance, the journal records each new disk’s devid, so a new disk that is gone then binds by it.

Scripted recovery

For unattended recovery (e.g. from a remote script):

echo "my-passphrase" | sudo braid recover --passphrase-stdin

Or with a passphrase file:

sudo braid recover --passphrase-file /path/to/passphrase

Recover for a replace journal when the pool is already mounted

Symptom: sudo braid recover exits with recover refuses to probe an already-mounted pool when the journal records a replace ... and instructs you to run braid lock first.

Cause: The pool was mounted by something other than braid recover itself (typically a manual cryptsetup open + mount after a crash, since braid unlock and braid-auto-unlock.service both refuse to mount when a pending-op journal exists). For a replace journal, the kernel may have resumed an interrupted dev_replace on that mount session, leaving stale in-memory device state that recover cannot distinguish from real topology. The cycle that scrubs this state needs to unmount and remount, which is unsafe on a mount recover does not own.

Steps

sudo braid lock      # works with a journal present -- no pending-op preflight
sudo braid recover   # opens its own mount and runs the relock cycle

braid lock unmounts the pool and closes the LUKS mappers. braid recover then opens a fresh mount session, finishes any in-progress kernel dev_replace, and runs the umount-and-remount cycle that clears stale btrfs_fs_devices – the standard happy path for replace recovery.

Unlock refused by a foreign or mismatched disk

Symptom: braid unlock exits with LUKS UUID mismatch. A disk at a recorded by-id slot reports a LUKS UUID that differs from the one in pool.json; the error names the disk, its by-id path, and the expected vs found UUID.

Cause: The disk was swapped, cloned, or reformatted out of band, so its LUKS identity no longer matches the recorded member. This is a hard refusal during probing, before any mapper opens. --allow-degraded does not bypass it – that flag only covers missing disks, and this disk is present.

If the swap was unintended

Detach the foreign disk and reattach the original. braid unlock then succeeds.

If the swap was intentional

braid replace requires the pool mounted, but the present mismatched disk blocks the mount. Make the slot read as missing first, then replace:

1. Detach the foreign disk so the member reads as absent.
2. Mount the pool degraded:

   sudo braid unlock --allow-degraded
   

3. Replace the now-missing member following Missing disk -> Option A: Replace the disk. braid replace prepares its own --new disk; see braid replace for how it handles a disk that already carries a LUKS header.

See also Out-of-band reformat during recovery for the same identity check on the braid recover path (a different trigger).

Missing disk (drive failure)

Symptom: braid status shows a device as missing. The pool may be mounted degraded or may refuse to mount.

Unlock with a missing disk

If the pool is not mounted:

sudo braid unlock --allow-degraded

This mounts the pool in degraded mode. All data is still accessible (btrfs RAID1 keeps a copy on the surviving disk(s)), but the pool is running with reduced redundancy until you replace the dead drive.

When the missing disk comes back

If the disk was only unplugged or came loose, plug it back in. btrfs does not take a returning disk back into a mounted pool, and braid add refuses a disk pool.json already lists as a member that is not live. Close and reopen the pool so the disk rejoins, then check its health:

sudo braid lock
sudo braid unlock
sudo braid doctor

Add --allow-degraded to braid unlock if another member is still missing. If the pool wrote data as single while the disk was away, braid doctor reports it and gives the balance that restores RAID1.

Hot-unplug while pool is mounted

If a drive is physically disconnected while the pool is mounted, its LUKS mapper can remain open with cryptsetup status reporting device: (null). btrfs continues to list the devid but has not yet promoted it to MISSING. braid status reports the devid – it contributes to missing_count and appears in missing_devids – but braid remove-missing --missing-id N and braid replace (with or without --missing-id) refuse the devid because they only act on btrfs-authoritative MISSING entries. braid remove-missing also refuses every other devid while this disk stays unpromoted, because btrfs still writes to it and could put relocated data on a disk that is gone. braid remove refuses the disk too and prints the steps below; braid doctor prints them as well.

Re-plugging the disk alone never reopens its mapper: it stays at device: (null) until it is closed and reopened. To make progress:

1. Check whether the disk is truly gone or only came loose.
2. If the disk is back (re-seated or re-plugged), close and reopen the mapper. braid unlock reopens it through the stable /dev/disk/by-id/ path, wherever the kernel re-attached the disk:

   sudo braid lock
   sudo braid unlock
   

   Add --allow-degraded to braid unlock if another member is MISSING.
3. If the disk is truly gone, relock and re-unlock the pool degraded so btrfs re-evaluates membership and promotes the devid:

   sudo braid lock
   sudo braid unlock --allow-degraded
   

   Re-run braid status – the devid should now appear as authoritatively MISSING – then retry braid remove-missing or braid replace.

A reboot in place of braid lock also closes the mapper. If an interrupted operation left pending-op.json, braid unlock refuses: use braid recover in its place (with --allow-degraded if a disk is gone).

Option A: Replace the disk

Replaces the dead disk with a new one, rebuilding data from surviving copies:

sudo braid replace --old toshiba2 \
  --new toshiba4=/dev/disk/by-id/ata-NEW_DRIVE_SERIAL

--old identifies the missing member. If you want to cross-check the btrfs devid from braid status, add --missing-id after the required args:

sudo braid replace --old toshiba2 \
  --new toshiba4=/dev/disk/by-id/ata-NEW_DRIVE_SERIAL \
  --missing-id 3

Replace runs btrfs replace start -B under the hood. braid replace is a long-running online operation: the command waits in the foreground and shows progress while the pool remains usable. It can take hours for large drives, so run it from a shell you can leave open (or a tmux/screen session). From another shell, braid status and braid tui can show progress independently.

Option B: Remove the missing device

Forgets the dead device without rebuilding data:

# Find the missing device's btrfs devid from braid status
sudo braid remove-missing --missing-id 3

Use this when you do not have a replacement disk. The pool continues with fewer disks and reduced capacity. Data that was only on the dead drive is lost (but in RAID1, all data has a second copy on another drive).

When this clears the last missing device and 2+ disks remain, remove-missing blocks on a follow-up soft RAID1 balance to restore redundancy on chunks written as single during degraded operation. You will see [wait] pool: restoring RAID1 redundancy... then [ok] pool: RAID1 redundancy restored before the command returns. The wait scales with how much data was written while degraded: an idle pool finishes in seconds, while a pool written to heavily during degraded mode can take longer. A sleep inhibitor is held for the entire operation. See braid remove-missing for the full sequence.

On a 2-disk pool, the dead disk was the only other copy, so remove-missing collapses the pool onto the present disk: it converts to single data and DUP metadata there, then removes the missing devid. The prompt warns Pool will have 1 disk -- no RAID1 redundancy, and the present disk must have room for all the data. The pool then has no redundancy until you braid add another disk.

Verify:

sudo braid status

A successful result shows no missing devices and no single profile rows for data or metadata.

Choosing between replace and remove-missing

Degraded mount

A degraded mount means at least one pool disk is missing. The pool is usable but the pool is running with reduced redundancy on the missing device’s share of data.

When degraded mounts happen

• braid unlock --allow-degraded – explicit request
• braid recover --allow-degraded – recovery with missing devices
• braid.autoUnlock.allowDegraded = true – auto-unlock config

Risks

• Reduced redundancy – the pool is short the missing device’s mirror copy of existing data, and on 2-disk pools new writes are allocated as single-profile chunks. A further drive failure could lose data.
• No self-healing – btrfs cannot repair corrupted blocks from a redundant copy if the copy was on the missing device.

Resolution

Replace the missing disk as soon as possible:

sudo braid replace --old <missing-name> \
  --new <new-name>=/dev/disk/by-id/<new-drive>

After replace completes, the pool is fully redundant again.

Recovery decision tree

braid command fails
├── "pending operation" error
│   └── braid recover [--allow-degraded]
├── pool.json missing
│   └── braid discover --write → braid unlock
├── "LUKS UUID mismatch" error
│   └── see "Unlock refused by a foreign or mismatched disk"
├── missing device / won't mount
│   ├── braid unlock --allow-degraded
│   └── then: braid replace or braid remove-missing
└── something else
    └── braid doctor → check troubleshooting guide

State files reference

All state lives under /var/lib/braid/:

Related

• Troubleshooting – symptom-oriented quick fixes
• NixOS configuration – autoUnlock.allowDegraded and other options