# Qualification supervisor operations

The qualification supervisor has two independent dead-man surfaces:

- the internal `qualification-watchdog` Compose worker evaluates durable
  snapshots, worker heartbeats, and leases;
- the user-level `sdq-qualification-observer.service` captures exact read-only
  controller status and host service state every 5–60 seconds.

Each observer cycle records a distinct `qualification-host-observer` heartbeat
and evaluates the application watchdog before controller access. The
application watchdog requires that heartbeat. This mutual dependency makes
either supervisor's sustained loss a durable UI and notification event, while
container and systemd restart policies remain the first-line recovery path.

The observer is read-only. Its script enables only the exact controller's
read-only adapter for one bounded capture, opens one tagged firewall rule, and
removes that rule on success or failure. It never enables the HPT2 command,
programmer, initialization, scrub, raw-media, or power adapters.

The user unit deliberately does not use systemd filesystem or
`NoNewPrivileges` sandboxing: those user-namespace controls remap the setuid
`sudo` binary and prevent the two required, policy-scoped host operations.
Privilege remains bounded by sudo policy plus the script's controller-role,
endpoint-port, Compose-service, nftables-tag, and cleanup checks.

Host-specific values belong in
`~/.config/sd-qualifier/qualification-supervisor.env`, created from the checked-in
example. The deployed release path and rendered service unit must be bound by
the reviewed deployment manifest. The manifest, database backup marker, and
controller registry remain protected read-only inputs to application
containers.

`SDQ_SUPERVISOR_EXPECTED_SERVICE_SEQUENCE` lists the ordered user services
that own the current campaign chain. Each cycle selects the first active,
failed, or missing service; it skips only inactive predecessors whose result is
`success`. This prevents an always-running observer from hiding a failed
matrix runner or a missing dependent transition. A single
`SDQ_SUPERVISOR_EXPECTED_SERVICE` remains a backward-compatible fallback.

`SDQ_SUPERVISOR_COMPOSE_ENV_FILE` and
`SDQ_SUPERVISOR_COMPOSE_OVERRIDE` must identify the existing protected runtime
environment and the exact release override. The observer refuses to run when
either file is absent, preventing fallback to fixture credentials or a mutable
default image tag.

`Restart=always` restarts the observer after a process failure. If capture or
restart remains unsuccessful, snapshot age expires and the independent Compose
watchdog creates a durable intervention. The web UI withholds forecasts until
controller, orchestration, and database truth agree again.

Never replay a Start or Stop to repair monitoring. Reconcile an unknown command
outcome read-only. Changing run, controller role, CID, boot witness, firmware,
or allowed state-changing scope requires a new reviewed authorization.
