# Watchdog Stage 1 Implementation and Fixture Gate — 2026-08-09

Status: PASS; supervisor source and first fixture gate satisfied without deployment or controller access
Type: evidence
Updated: 2026-08-09
Next Action: execute Stage 2 controller-free failure-detection qualification; keep the supervisor undeployed, Controller A held, and Controller B not started
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/wp/completed/wp-watchdog-gated-controller-a-recovery-and-controller-b-start.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

Scope: source implementation and fixture qualification only. No live
qualification service was deployed or restarted, no migration was applied to
the live database, no controller endpoint was configured or contacted, no
controller command was issued, Controller A remained held, and Controller B
was not started.

## Versioned source identity

- Repository: `jong@srv-rpi00.local:/home/jong/dev/sd_qualification-watchdog-stage1`
- Branch: `codex/qualification-watchdog-stage1`
- Base: `origin/main` at `abca3b08c788e467f6be7ff5ad783d106ba34b4c`
- Commit: `71cd342cfba8ee044cee5c393b80b08110bfe136`
- Commit subject: `Add qualification watchdog and fail-closed supervision`
- Fixture image: `sdq-watchdog-stage1-test`
- Fixture image ID: `sha256:32204e793208231f38ae2f21870c4b87df22479a135a1ed20dae3122a260abf2`
- The isolated Pi worktree was clean after the commit. The source was not
  pushed and was not deployed.

## Implemented gate surface

- Append-only `sdq-qualification-supervisor-snapshot/v1` three-plane records
  and a durable per-campaign supervisor projection.
- Reconciliation of run/controller/card/boot/firmware identity, monotonic
  controller heartbeat and attempted/committed/verified counters, lane and
  latched-failure state, application/database heartbeat, observer/service
  state, unknown command outcomes, evidence/backup/disk health, notification
  delivery, and stuck/orphaned process counts.
- Profile-provided no-progress lease bounded by a server maximum. The lease
  renews only when counters advance; the hard-review deadline can move earlier
  but never later.
- Durable transition and recovery events plus deduplicated `RUN_BLOCKED`
  interventions using `SAFE_STATE_UNCONFIRMED`. The requested action explicitly
  prohibits replaying or inferring controller commands.
- A persistent `qualification-watchdog` Compose worker with inherited
  `restart: unless-stopped`, an internal-only network, protected configuration
  mounted read-only, and its own `WorkerHeartbeat` dead-man signal.
- Dashboard, campaign, and exact-card banners. Blocked supervision overrides
  the false-green automated operator action and withholds open forecasts.
- A hash-verified deployment-manifest contract binding source, images,
  orchestration scripts/units, database migration/schema identity, and
  qualification profiles. The real production manifest remains a Stage 3
  deployment artifact; the checked-in file is fixture/example data only.

## Verification

All verification used the isolated source commit and no live volumes or
controller networks.

| Check | Result |
| --- | --- |
| Django system check and migration application | PASS |
| `makemigrations --check --dry-run` | PASS — no model drift |
| Focused watchdog, manifest, and UI tests | PASS — 16/16 |
| Compose isolation/restart-policy tests | PASS — 7/7 |
| `docker-compose config --quiet` | PASS |
| Full disposable SQLite regression | PASS — 276/276 in 78.252 s |
| Full disposable PostgreSQL 17 regression | PASS — 276/276 in 82.552 s |

The focused set proves healthy three-plane operation, planned hold, observer
stop while the controller remains active, stale database heartbeat with a live
controller, missing snapshots, CID mismatch, counter regression, unknown
Start/Stop outcome, progress renewal, no-progress expiry, non-renewable hard
review, durable notification, recovery event behavior, snapshot immutability,
watchdog dead-man heartbeat, prominent web status, and forecast suppression.

The first full-suite invocation used the image default read-only evidence path
and produced two environmental permission errors in existing HPT2 tests. It
was rerun with the intended disposable writable fixture root
`SDQ_EVIDENCE_ROOT=/tmp/evidence`; both complete database-engine regressions
then passed. No source change was made to conceal those fixture-path errors.

## Cleanup and next gate

The exact disposable PostgreSQL container
`sdq-watchdog-stage1-postgres` and network `sdq-watchdog-stage1-net` were
identity-checked, removed, and verified absent. No test container remains
running. The local fixture image remains as a reproducible Stage 1 artifact.

Stage 2 is next: complete the controller-free fault matrix, including watchdog
termination/restart, normal transitions without false alarms, notification
provider outage/retry/acceptance, worker/database restart behavior, deployment
and storage faults, and synthetic independent A/B concurrency. Stage 2 does
not authorize deployment or controller access.
