Device Disappearance States

When a physical drive disappears from a btrfs pool (hot-unplug, cable failure, drive death), the system passes through several states depending on how far the failure has progressed and whether the LUKS mapper is still open. Each state produces different output from btrfs filesystem show, btrfs device stats, and cryptsetup status — and braid must handle each combination correctly.

This mapping is not derivable from reading braid’s code or btrfs docs alone — it requires cross-tool knowledge that’s easy to get wrong.

State Table

Empirical note: SATA hot-unplug on real hardware enters Null-underlying immediately and stays there for at least 5 minutes without I/O pressure. We have not yet observed the MISSING-with-path state in practice. See real-world/sata-hot-unplug.md for full test results.

Healthy

Normal operation. Physical drive is present, LUKS mapper is open and points to the underlying block device, btrfs sees the device.

Null-underlying

Hot-unplug while mounted. The LUKS mapper (/dev/mapper/braid-X) is still open in device-mapper, but the backing block device has vanished. cryptsetup status reports device: (null). btrfs still sees the mapper path — it doesn’t know the physical drive is gone until I/O fails.

braid handles this correctly: probe_pool detects the (null) device and records it in pool.null_underlying. monitor’s own probe, probe_pool_alerts, records only its devid (null_underlying_devids), and monitor includes that devid in the alert-local missing set (AlertDevids.missing). The stats row reports both the mapper path and the devid; the alert pipeline pairs by devid directly. braid doctor’s pool_missing_devices check also keys on missing_count (which counts null-underlying members), so it warns here rather than reporting a green “no missing devices” — and routes the operator to the re-seat or promote-then-replace recovery (braid replace and remove-missing refuse a null-underlying devid until btrfs promotes it to MISSING).

btrfs also keeps the member writable. btrfs device usage still prints its mapper path and full size: btrfs-progs reads the size with a BLKGETSIZE64 ioctl on the mapper, which returns the dm table’s size without touching the backing disk (btrfs-progs v6.19.1, common/device-utils.c (device_get_partition_size), called from cmds/filesystem-usage.c (load_device_info)). The kernel chunk allocator still places new chunk copies on it. So remove-missing refuses every target while any other member is in this state (cli/src/types.rs#PoolState::hot_unplugged_not_missing); relocating chunks would put copies on a disk that is gone. See ADR 012.

Post-UUID-identity rule: when a mapper is null-underlying, the backing disk’s LUKS header is gone, but the mapper’s dm UUID (/sys/block/dm-N/dm/uuid) still records the LUKS UUID cryptsetup opened it with. braid binds the live mapper back to membership by that UUID (cli/src/dm_identity.rs#mapper_luks_uuid), as it does for a mapper whose disk is present. So status’s compact Drives: row, replace, and every recovery path take a null-underlying member’s devid from its mapper, not from pool.json, which may hold none or a stale one. Recovery does this through one binding pass (cli/src/membership.rs#bind_live_pool) and keeps the member in pool.json. Persisted DiskMember.devid binds only a device btrfs reports MISSING with no open mapper (cli/src/types.rs#PoolState::devid_only_missing); it is prior-binding state, not display authority, and status output still uses live btrfs stats for displayed devids.

MISSING with path

btrfs has registered the device as missing, but still remembers which mapper path it had. btrfs filesystem show appends MISSING to the path. The parser puts the devid into missing_devids but discards the path. probe_pool never processes this device (it only iterates show.devices), so it doesn’t appear in pool.devices or pool.null_underlying.

Handling: btrfs device stats rows always carry a mandatory devid field, so the alert pipeline identifies the row by devid regardless of which path string btrfs reports ([/dev/mapper/X] or [devid:N]). The MissingDevice alert is generated independently from missing_devids. Rows for alert-local missing devids are skipped for BtrfsDeviceErrors, so the monitor latches no counts for them and braid ack commits none. If the member returns, the next monitor cycle alerts once on the counts it logged, including the I/O that failed as it went, and the next ack commits them (see ADR 014).

This is the one state where membership correlation falls back to devid: btrfs reports the missing device only by devid, so braid resolves the member whose persisted DiskMember.devid matches. It must not infer membership by parsing a mapper name or LUKS label.

Uncertainty: We haven’t empirically confirmed which path string btrfs device stats reports for a device in this state – the ?? in the table marks this. The answer no longer affects correctness (devid drives the lookup), but it would still be useful empirical data.

Fully gone

Device is completely absent — either the LUKS mapper was torn down, or the device was missing at mount time (degraded mount). btrfs filesystem show reports bare path MISSING (no mapper path). The pinned btrfs-progs renders the missing-device stats path as [devid:N] (cmds/device.c#print_device_stat_string); [<missing disk>] is an older btrfs rendering. braid does not depend on either string: the parser ignores the device field and keeps the row’s devid and counters.

At this point there is no mapper and no observable LUKS UUID. Mutating commands that target the missing device, such as remove-missing and missing-path replace, resolve the requested btrfs devid through UUID-keyed membership and fail closed if no persisted member carries that devid.

Transitions

The typical progression for a hot-unplug:

Healthy → Null-underlying → MISSING with path(?) → Fully gone

The transitions depend on timing, I/O activity, and whether the kernel tears down the LUKS mapper. A brief unplug-replug might only reach Null-underlying before recovering. A permanent removal eventually reaches Fully gone.

The transition from Null-underlying to MISSING with path is the least understood. It likely happens when btrfs attempts I/O on the device and gets errors, then marks it missing — but the mapper path is still in kernel memory so btrfs remembers it.

Code Pointers

• probe_pool: cli/src/probe.rs – builds pool.devices, pool.null_underlying, pool.missing_devids
• btrfs filesystem show parser: cli/src/parse/btrfs_filesystem_show.rs – filters MISSING devices from devices list
• btrfs device stats parser: cli/src/parse/btrfs_device_stats.rs – propagates devid as the btrfs-native stats row key and ignores the display-only device string
• alert computation: cli/src/alert.rs – compute_alert_state keys by dev.devid from the parsed stats row and skips alert-local missing devids for BtrfsDeviceErrors; commit_latched_evidence writes ack baselines by the latched cause’s devid