# Watchdog Stage 2 fault-matrix evidence — 2026-08-09

Status: PASS; controller-free failure detection and notification qualification complete
Type: evidence
Updated: 2026-08-09
Next Action: retain as the Stage 2 gate for the production-safe Stage 3 deployment
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/wp/completed/wp-watchdog-gated-controller-a-recovery-and-controller-b-start.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Scope

Stage 2 exercised disposable fixture services and synthetic campaign/controller
records only. Neither controller endpoint was contacted, no controller command
was created or published, Controller A remained held, and Controller B remained
at intake.

## Qualified supervisor behavior

- The supervisor projection is unique per campaign and controller role, so
  concurrent synthetic A/B observations, incidents, and notifications cannot
  overwrite one another.
- The in-container watchdog and host observer have independent heartbeats and
  detect the loss and recovery of the other.
- Normal progress, planned holds, planned stops, clean terminal results, and
  dependent stage transitions remain healthy.
- Missing/stopped observers, stale database or controller heartbeats, unknown
  command outcomes, identity or counter divergence, progress-lease expiry,
  hard-review expiry, latched failures, worker/storage/backup/disk faults,
  deployment divergence, and notification failures produce durable visible
  blocking state.
- Notification episodes are keyed to distinct transition events. A changed
  failure detail creates a new episode; unchanged repeated polls do not flood
  the outbox. Delivery failures remain visible and retryable.
- Fixture orchestration uses an isolated, disposable PostgreSQL volume name;
  no production volume was mounted during the matrix.

## Verification

| Check | Result |
| --- | --- |
| Focused watchdog/fault/notification suite | PASS |
| Watchdog held-down alert and recovery | PASS |
| Host-observer independent dead-man proof | PASS |
| Missing snapshot and service-divergence projection | PASS |
| Synthetic role-A/role-B independence | PASS |
| Real SMTP provider acceptance | PASS at `2026-08-09T20:03:15Z` |
| Complete disposable SQLite suite | PASS — 290/290 |
| Complete disposable PostgreSQL 17 suite | PASS — 290/290 |
| Migration drift check | PASS — none |

The complete supervisor implementation was committed as
`2441a7778ff4560fdf3665576b3ca0dc70d84e69` (`Complete qualification
watchdog supervision`). Exact-compose, bootstrap-heartbeat, and scoped host
observer corrections followed as `09d38c43`, `c2f8b721`, and final Stage 3
candidate `94768c9cef418260b16028357c24028495235e29`.

## Cleanup

The Stage 2 fixture containers were stopped and removed after Stage 4 live
verification. Their disposable fixture volume was retained; no production
container or volume was removed.
