Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

UPS Integration

Principles:

Context

A btrfs RAID1 pool tolerates clean shutdowns, but sudden power loss during active I/O – especially during a long-running btrfs replace, btrfs device remove, or post-add/remove balance – can leave the pool in a state that requires manual recovery. This is the same risk surface that decision 019 protects against for suspend/wake, but it cannot use the same control model. A sleep inhibitor actively blocks the operating system from suspending; braid cannot analogously block a UPS from running out of battery. The control model here is different: reject avoidable starts on battery up front, and prove journal recovery for the unavoidable mid-mutation case.

A UPS solves this only if the host cooperates. NUT (Network UPS Tools) is the standard Linux interface, and nixpkgs already provides a mature power.ups module that configures NUT declaratively – units, users, udev rules, killpower handling. braid’s job is not to reimplement that, but to layer opinionated policy on top so that enabling UPS support gives a home NAS three specific guarantees:

  1. Orderly shutdown before battery exhaustion for ordinary mounted operation.
  2. Preflight refusal to start pool-mutating commands unless the UPS reports verified utility power (OL).
  3. Live UPS state visible in braid ups status and the TUI; live UPS status is used for preflight safety and upsmon critical-state shutdown (normally OB + LB together, per reference/nut/clients/upsmon.c:1404).

The guarantees do not extend to “safe against any power loss.” A UPS firing LB during a mutation that started on AC still interrupts that mutation. Recovery for that case falls to the existing journal + braid recover path, and must be proven per mutation class by VM tests before this decision flips to Active.

“Just alert the user on low battery” is insufficient for guarantee (1): a prolonged outage with nobody present would still exhaust the battery during an active mount. The host must power off before battery exhaustion, because decision 018’s teardown sequence (braid-online.service ExecStop -> btrfs umount -> luks close) needs a non-trivial window of live power to complete cleanly.

Decision

Scope: standalone, USB, single-host

v1 supports one NUT-compatible UPS connected over USB to the NAS, monitored by the NAS itself. Not supported as first-class:

  • networked NUT (primary/secondary across multiple machines)
  • serial apcsmart, snmp-ups, or other non-USB drivers
  • multiple UPSes per host

An escape hatch (driver = "...", port = "...") exists for users whose UPS speaks a non-USB protocol, but braid does not guarantee correct behavior outside the USB path.

Rationale: USB UPSes cover the vast majority of home NAS deployments. Every non-USB topology adds configuration surface (network auth, SNMP community strings, serial port permissions) that braid would have to validate and test. Single-host standalone avoids the two-machine primary/secondary dance and its timing/credential complexity.

Wrap power.ups, do not reimplement NUT

The braid module sets power.ups.* values from its higher-level options. It does not write ups.conf, upsd.conf, or upsmon.conf directly, and does not define its own nut-* systemd units.

This is a deliberate departure from the pattern in modules/braid/fan-control.nix, which owns its unit because nixpkgs’ hddfancontrol module has concrete lifecycle bugs. The nixpkgs power.ups module has no equivalent known defect; reimplementing its surface would duplicate work and diverge over time.

Data source: shell out to upsc

The TUI and braid ups status command read UPS state by invoking upsc <name> and parsing its key/value output. This matches every other braid parser (btrfs, cryptsetup, lsblk, smartctl, smartd, hddfancontrol).

A parse_upsc module in cli/src/parse/ handles the parse, with stable and unstable golden fixtures in cli/tests/fixtures/. NUT (networkupstools) joins btrfs-progs and cryptsetup in the pinned parser-critical toolchain (see decision 010 and parser compatibility), with fixture refresh required on any nixpkgs bump that changes its pinned version. util-linux remains parser-covered but is host-provided under decision 010.

Pinning is load-bearing. This decision adds the braid.packages.nut override alongside btrfsProgs, cryptsetup, and utilLinux, defaulted to nixos-26.05’s pinned pkgs.nut. util-linux has since become host-provided while remaining overrideable; see decision 010. The module assigns this pin to power.ups.package and includes the same derivation in the CLI wrapper’s PATH so that upsc invoked from braid ups status resolves to the tested version rather than whatever the host’s system path provides. Decision 010 and principle 10 are updated in the same implementation to name NUT as parser-critical.

ups.status is parsed into an ordered, deduplicated list of flags (OL, OB, LB, CHRG, DISCHRG, RB, …), not an enum. Flags are stored in upsc emission order; membership and dedup give set semantics without imposing a sort. Display severity is derived from the combination; unknown tokens are preserved in the parsed model so that new NUT statuses do not silently disappear.

braid ups status defaults to a curated human-readable summary and supports --json for the typed parsed model plus braid’s severity verdict. Raw upsc passthrough is not exposed; users who want that can still run upsc directly.

The --json success shape preserves the typed parsed model at top level and adds a top-level severity: the shared verdict (cli/src/parse/types.rs#UpsSeverity) that every other UPS surface, mutation preflight included, also matches on. An empty or missing ups.status is its own status_missing verdict, still exit 0. Only "severity": "online" proves utility power; any other value, or a top-level .error (whose body carries no severity), means it is not proven, so a value added later fails closed. The status_flags array preserves first-seen ups.status token order across the human, --json, and TUI surfaces – braid imposes no sort of its own. Order is deterministic for a given UPS state (whitespace is normalized and repeated tokens collapse to first-seen); it is not a byte copy of the raw ups.status: line.

Rationale for the severity field: a verdict on the success body lets scripts take braid’s judgment instead of re-implementing the ladder from status_flags and drifting when it changes. It replaced the earlier "warning": "ups_status_empty" sentinel, which folded into the status_missing verdict, so UpsSeverity::classify is the only code that judges the empty case on every surface. This reverses plans/impl/2026-06-04-ups-status-json-trusted-healthy-contract.md#decision, which rejected a JSON verdict because it would fork the classifier into a third surface and scripts would still need the flags to tell OB from LB. Both reasons lapsed: plans/impl/2026-06-17-unify-ups-severity-classification.md made UpsSeverity the one classifier, so serializing it consumes that classifier instead of copying it, and the verdict separates on_battery from critical. Scripts still read status_flags to learn which flag fired, not to decide whether utility power is proven.

Shutdown-on-LB = systemctl poweroff

When NUT fires the low-battery (LB) event, upsmon runs systemctl poweroff. systemd’s standard shutdown sequence then unwinds braid-online.service (decision 018), which closes the btrfs mount and LUKS mappers. The host powers off via normal means before the UPS exhausts its battery.

This is not “alert only.” The host genuinely shuts down, because the only safe state during a prolonged power outage is off. An alert-only policy would require the user to react in time, which defeats the point of unattended operation.

Early shutdown: on-battery timer and charge floor

Firmware LB is the latest possible moment to shut down – often about two minutes of estimated runtime left, after a deep discharge, on a battery whose estimate drifts as it ages. So braid also shuts down early: braid.ups.shutdownAfter (seconds on battery) and braid.ups.shutdownBelowCharge (percent), whichever comes first, both on by default. LB stays the backstop and is unchanged.

braid-ups-guard.service runs braid ups guard (cli/src/ups_guard.rs). It polls upsc and, when a trigger holds, runs upsmon -c fsd. It never powers the host off itself: upsmon sets FSD, writes POWERDOWNFLAG, and runs SHUTDOWNCMD, so the shutdown, the braid-online unwinding, and ups-killpower are the same path as LB. The on-battery clock starts on the first OB reading and stops only on an online verdict from UpsSeverity; a failed or unproven reading changes nothing. The clock is a CLOCK_BOOTTIME value in the unit’s preserved runtime directory, so it survives a guard restart, counts time spent suspended, and resets at boot. The guard runs as the upsmon user with no capabilities: signalling upsmon needs only the same uid. It is not pool-bound. Its policy reaches it through /etc/braid/config.json.

A dead guard silently drops the host back to LB, the same class of fault as an inactive braid-online, so braid doctor fails when a trigger is configured and the guard is not running. It warns when shutdownBelowCharge is set but the UPS reports no battery.charge, since the floor can then never fire (cli/src/doctor.rs#check_ups_guard_active, cli/src/doctor.rs#check_ups_charge_floor).

Rejected: NUT’s ignorelb with battery.*.low overrides, which drops the device’s own LB and stops the driver if the UPS stops reporting charge or runtime; upssched timers, which see events but not charge and would make braid own NOTIFYFLAG; writing firmware thresholds with upsrw, which depends on the device and lives outside the NixOS config; and a guard that runs systemctl poweroff itself, which skips POWERDOWNFLAG so the UPS never cycles power. Full rationale: plans/impl/2026-10-22-1122-ups-on-battery-shutdown-policy.md.

Reject pool-mutating commands unless UPS reports utility power (preflight hygiene only)

When braid.ups.enable = true, braid add, braid remove, braid remove-missing, and braid replace query UPS status at preflight and refuse with a Validation-shaped error unless the UPS status can be trusted as explicitly on utility power. The check is fail-closed: it refuses on upsc invocation or query failure (dead upsd, unknown UPS name, or exec failure), an empty or missing ups.status, any critical flag (LB, TESTFAIL, COMMBAD, FSD – the same set the TUI paints red), on-battery (OB), or any status set missing OL. Known non-critical advisory states such as OL RB, and unknown tokens co-present with OL and no known blocker, still pass because utility power is explicitly present. The check sits alongside the existing preflight checks, before any journal write.

This is preflight hygiene, not a mutation-window guarantee. It narrows the surface that journal recovery must cover by rejecting the easy case – “user starts braid replace while the power is already out” – but it cannot and does not prevent LB from firing mid-mutation on work that started on AC. Mid-mutation power loss is handled by the existing journal + braid recover path; see the recovery-proof obligation in Open Questions.

This is not the power-side equivalent of decision 019’s sleep inhibitor. A sleep inhibitor actively blocks suspend for the duration of the mutation window; braid cannot analogously block UPS-driven shutdown, because the UPS is dying and no amount of inhibiting changes that. Instead, the contract is: reject the avoidable case up front, and rely on recovery for the unavoidable case.

Alert-model integration is deferred

Integrating UPS events into the shared AlertState / AlertCause model (decision 014) is deferred to a future ADR. Decision 014 guarantees “alerts stay latched until braid ack” – the right shape for event-driven causes (disk errors, smartd), but wrong for live-state conditions like OB / LB (users expect those to clear when the UPS returns to OL). Reconciling that requires splitting AlertCause by persistence semantics (LatchedUntilAck vs. ActiveWhileConditionHolds) and updating merge_into_latch, ack, status, and the alert-test matrix. That is a core-invariant change that deserves its own ADR; smuggling it into UPS v1 would conflate two distinct concerns.

v1 therefore surfaces UPS state only through braid ups status and the TUI. Operators who are not actively watching those surfaces will not see on-battery or comms-loss conditions asynchronously in v1. This is a known gap until the follow-up ADR lands.

braid-online becomes safety-critical under UPS

mark_online (cli/src/online_state.rs) warns and exits successfully when systemctl start braid-online.service fails after a successful unlock/add/recover. When UPS support is enabled, this silent-degradation path is unsafe: the user believes LB will trigger a clean shutdown, but without braid-online.service active, its ExecStop does not run and LUKS close is not guaranteed to complete before power dies.

braid doctor and the TUI flag “pool mounted but braid-online inactive” as a high-severity configuration fault whenever UPS support is enabled. mark_online’s warn-and-continue behavior otherwise remains unchanged; the UPS path adds a new detector, it does not change the underlying unlock sequence. Under systemd_lifecycle = false (CLI-only), the lifecycle path is skipped entirely; the UPS-safety detector fires only when systemd_lifecycle = true and UPS support is enabled.

Upsmon credential lifecycle

NUT requires upsmon to authenticate to upsd even in single-host standalone mode. The credential lives at /var/lib/braid/upsmon.pass with mode 0600, owned by root, outside the Nix store.

Generation: a oneshot braid-ups-secrets.service creates the file if absent with a random token (e.g. head -c 24 /dev/urandom | base64) and exits. The oneshot is wired with before = [ "upsd.service" "upsmon.service" ] and requiredBy = [ "upsd.service" "upsmon.service" ] (the actual nixpkgs power.ups unit names), so upsd and upsmon hard-fail to start if secret creation fails rather than racing it. systemd.tmpfiles rules ensure /var/lib/braid/ exists with correct ownership before the oneshot runs. The file is stable across rebuilds; regeneration happens only on explicit deletion. No rotation is performed because the scope is loopback upsmon<->upsd on a single host.

Reference: the rendered NUT configs consume the file via power.ups.users.<name>.passwordFile and power.ups.upsmon.monitor.<name>.passwordFile (not inline passwords), so the token never enters the Nix store or nix-store --query output.

Proposed config surface

braid = {
  enable = true;

  ups = {
    enable = true;
    name = "ups";               # identifier used by upsd and upsc
    driver = "usbhid-ups";      # USB default; covers the vast majority of UPSes
    port = "auto";              # usbhid-ups's standard "find the device" value
    shutdownAfter = 300;        # seconds on battery before early shutdown; null = off
    shutdownBelowCharge = 50;   # percent charge floor on battery; null = off
  };
};

Defaults applied internally, not surfaced as options in v1:

  • standalone mode (upsd + upsmon on the same host, no network monitors)
  • SHUTDOWNCMD = systemctl poweroff
  • upsmon credentials per “Upsmon credential lifecycle”

Note: NOTIFYCMD is intentionally not configured in v1 – alert-model integration is deferred (see “Alert-model integration is deferred” above). The early-shutdown guard polls instead of using NOTIFYCMD, so users’ own NUT notification setups stay theirs.

The configured name and shutdown policy are also written to /etc/braid/config.json so that braid ups status, the TUI, and the guard do not have to guess which UPS to query.

Deferred

  • networked NUT (primary/secondary across hosts)
  • non-USB drivers as first-class support (work via escape hatch, not tested)
  • pre-shutdown grace window with braid ups abort-shutdown
  • battery-age reminders driven by battery.mfr.date + the RB status flag
  • multi-UPS per host
  • UPS-triggered automatic pause of running balance (scrub is cancelled on shutdown and resumed on next pool activation; crash-paused owed RAID1 balance now fails closed in recover while idle/no-paused owed replay still runs)

Resolved questions

Each of these blocked the flip from Draft to Active. All three are now closed by VM tests committed in tests/module/.

  1. Recovery-proof for mid-mutation power loss (primary blocker). Resolved by the four VM tests in plans/impl/2026-04-21-forced-shutdown-recovery-proof.md’s matrix: ups-lb-during-replace, ups-lb-during-remove, ups-lb-during-remove-missing, and ups-lb-during-balanced-add. Each fires OB LB via upsrw while a different mutation class is in flight, lets systemctl poweroff run, reboots the VM, and runs braid recover. The idle/no-paused recovery path still asserts the post-recover state matches what the original mutation would have produced – including no orphaned LUKS mappers, no MISSING btrfs entries, no remaining single-profile chunks where RAID1 was intended, and a cleared pending-op.json. The crash-paused owed RAID1 subcase is intentionally narrower: ups-lb-during-remove-missing and the paused branch of ups-lb-during-balanced-add now assert that recover preserves pending-op.json, leaves single-profile chunks visible, and asks for manual btrfs inspection instead of replaying a balance. The Pre-M11 audit also surfaced two cli/src/recover.rs gaps that the same plan landed before the matrix ran: pool_resize_device is now replayed for OpKind::Replace, and the owed RAID1 balance is replayed for OpKind::Add, OpKind::RemoveMissing, and OpKind::Replace only when btrfs balance status is idle – for an add into an existing pool the same hard convert live add runs (preceded by a soft pass when the cut-off convert left mixed chunk profiles, which a hard convert alone hits as ENOSPC; see balance-soft), otherwise the soft restore; see balance-soft for the underflow rationale behind the fail-closed branch.
  2. Shutdown ordering for ordinary mounted operation. Resolved by tests/module/ups-lb-clean-shutdown.{nix,py} (Plan 1’s M7). The VM test mounts an idle pool, fires OB LB via upsrw, and asserts braid-online.service’s ExecStop completes (and is not killed by TimeoutStopSec) before poweroff. The default TimeoutStopSec = 5min is sufficient for a single-disk pool; larger pools should retain that headroom.
  3. Battery-low threshold. Resolved by not depending on it. Plan 1’s M7 (ups-lb-clean-shutdown) showed the driver’s LB, at upstream upsmon timings (POLLFREQ/POLLFREQALERT/FINALDELAY = 5/5/5, via tests/module/lib/ups-fixture.nix with upsmonTimings = null), leaves enough time for a clean shutdown of an idle pool. But LB fires late and on the firmware’s terms, and battery.runtime.low / battery.charge.low set in ups.conf only publish a value: NUT raises LB from them only under ignorelb (reference/nut/drivers/dstate.c#status_commit). Early shutdown is therefore braid’s own policy – see “Early shutdown: on-battery timer and charge floor” above, proven by tests/module/ups-early-shutdown.{nix,py} – and LB is only the backstop.

Consequences

  • enabling UPS support is one line of Nix, plus two optional strings for non-default drivers
  • for ordinary mounted operation, the host powers off cleanly without user intervention: early, after shutdownAfter on battery or below shutdownBelowCharge, and on low battery at the latest
  • pool-mutating commands refuse to start unless utility power (OL) is verified, narrowing the journal-recovery surface to the mid-mutation case
  • mid-mutation power loss is a supported recovery case, not a guarantee: braid recover is load-bearing for replace / remove / remove-missing / balanced add interrupted by LB-driven shutdown, and VM tests prove both the idle/no-paused success path and the crash-paused owed RAID1 fail-closed path
  • live UPS state is visible in braid ups status and the TUI; users not actively watching those surfaces do not get asynchronous notifications in v1 (alert-model integration deferred to a future ADR)
  • NUT joins btrfs-progs and cryptsetup as a pinned parser-critical tool; nixpkgs bumps touching networkupstools trigger the same fixture-refresh obligation as the other pinned parser tools
  • the existing braid-online.service lifecycle (decision 018) is load-bearing under UPS; its failure mode is no longer acceptable silent degradation and braid doctor reflects that