# Watchdog-gated Controller A recovery and Controller B start Workpackage

Status: complete
Type: workpackage
Updated: 2026-08-09
Next Action: preserve the completed watchdog gate; admit later Controller B cases separately while Controller A continues under the production watchdog
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md
Canonical Truth: `jong@srv-rpi00.local:~/dev/sd_qualification/README.md`, its current `docs/`, and the live controller/database evidence captured at each gate

## Stage 0 checkpoint — PASS

Stage 0 completed on 2026-08-09 without a controller-state command. The
database backup and disposable restore passed; exact read-only A identity,
held-run status, and B identity passed; orchestration, services, deployment,
storage, notifications, and Windows process state were inventoried; cleanup
restored the default firewall with no target-tool container remaining.

Controller A remains held at `44641` with exact unchanged run/CID/boot identity
and zero recorded failures. Controller B has a distinct installed KEXIN CID but
remains uninitialized, at intake, with no run or campaign CID record. Evidence:
`work/evidence/watchdog-stage-0-preservation-baseline-20260809.md`.

## Stage 1 checkpoint — PASS

Stage 1 completed on 2026-08-09 in an isolated Pi worktree without deployment,
live database migration, controller access, or controller commands. The
versioned supervisor commit is
`71cd342cfba8ee044cee5c393b80b08110bfe136`. It provides append-only
three-plane observations, progress-aware and hard-review leases, fail-closed
events/interventions, visible web blocking and forecast suppression, a
restart-supervised internal-only Compose worker, dead-man health, and a
hash-bound deployment manifest contract.

Focused watchdog/UI/manifest tests passed 16/16, Compose policy tests passed
7/7, and the complete 276-test application suite passed on both disposable
SQLite and PostgreSQL 17. Evidence:
`work/evidence/watchdog-stage-1-implementation-and-fixture-gate-20260809.md`.
The source is committed but unpushed and undeployed. Stage 2 remains entirely
controller-free; A remains held and B remains not started.

## Stage 2 checkpoint — PASS

Stage 2 completed on 2026-08-09 without controller access. The fixture matrix
proved independent A/B projections, normal/planned transitions, missing and
stale observations, service and worker divergence, mutual dead-man behavior,
storage/backup/disk/deployment failures, notification outage/retry, distinct
transition episodes, and real SMTP provider acceptance. The complete suite
passed `290/290` on disposable SQLite and PostgreSQL 17. Evidence:
`work/evidence/watchdog-stage-2-fault-matrix-20260809.md`.

## Stage 3 checkpoint — PASS

Stage 3 deployed exact production release `qualification-watchdog-94768c9`
after a restore-verified backup and migration. The deployment manifest,
workers, storage, clock, bundle/profile, observer, and watchdog gates passed;
state-changing adapters remained disabled. A forced watchdog-process crash
restarted and recovered without changing A, and the four exact stale Windows
campaign processes were retired. No controller command was issued. Evidence:
`work/evidence/watchdog-stage-3-production-deployment-20260809.md`.

## Stage 4 checkpoint — PASS

Stage 4 completed on 2026-08-09. A fresh exact A snapshot matched the preserved
run, CID, boot, firmware, checkpoint, counters, lanes, and zero-failure state.
The existing Start was not replayed. One exact Stop was acknowledged, four
stable clean terminal observations supported finalization, and the result was
preserved as the first S4 `44640 / AFTER` extended observation while the base
classifier correctly reported its S3-shaped pre-Stop boundary.

Audit-selected replicate 2 run `de340af2-d505-495e-a14f-5159e55b1633` then
received exactly one acknowledged Start. A live false-positive lane-transition
finding was corrected in commit `6e26d778852ae24094af5939cb84936c639470f2`;
focused tests passed `31/31`, the full suite passed `295/295`, and exact image
`sha256:14094d1659c54a9ccf9738022b5d84498e1ae0a021bec0fb16652042580e40ad`
is deployed. Four consecutive new-release snapshots were healthy while A's
verified counter advanced `1070 -> 1124 -> 1179 -> 1232`. Controller B remains
at intake and was not started. Evidence:
`work/evidence/watchdog-stage-4-controller-a-recovery-resume-20260809.md`.

## Stage 5 checkpoint — PASS

Stage 5 completed on 2026-08-09 without Controller B access. Accepted ADR
`work/adr/adr-controller-b-parallel-qualification-entry-policy.md` replaces
the old A-local-matrix-complete scheduling rule with separate B Preflight and
Start decisions. The protected policy, exact evidence/resource identities,
and A 3-snapshot/60-second proving window pass. B admission is gated before
observation, and B Start is gated before intent and again before publication.

Focused tests passed `11/11`; full exact-image suites passed `299/299` on
SQLite and PostgreSQL 17. Production now runs source `dc492e4`, exact image
`sha256:6543e1ea2f1515e571d12e8329fb447663262883d8817c379ec8268441003db7`,
and deployment manifest `b1d3e06a...9094f`. Campaign event 251 records
`preflight_allowed=true` and `controller_b_execution_allowed=false` with every
remaining B blocker. A remains healthy and progressing; B remains `INTAKE`
with zero active runs/operations. Evidence:
`work/evidence/watchdog-stage-5-controller-b-parallelization-policy-20260809.md`.

## Stage 6 checkpoint — PASS

The owner power-off installation attestation is immutable evidence `1842` and
event `254`. B's full-address scrub passed all `15,728,640` sectors with zero
failure, recovery, or identity counts; the same-CID post-scrub observation and
HPT media-2 initialization passed; and exact S0 run
`ffb59651-0bb0-4d9b-a729-0099cc0f5690` is admitted at
`HPT2_MEASUREMENT_READY` without target access.

Commit `5b80fda` corrects lifecycle-policy matching across later same-card CID
captures. Focused tests pass `23/23`; full SQLite and PostgreSQL 17 suites pass
`303/303`. Production image `sha256:931cfce...d950` and manifest
`2fac279c...fe88` are healthy. Event `260` records
`controller_b_execution_allowed=true`; the UI names the exact S0 Start action.
B still has zero authorizations and zero HPT2 command operations. Evidence:
`work/evidence/watchdog-stage-6-controller-b-preflight-complete-20260809.md`.

## Stage 7 checkpoint — PASS

Fresh exact authority, final same-identity B/CID observation, and three
new-manifest healthy A snapshots passed before publication. Commit `b14dc8e`
corrects the exact prepared-operation publication recheck without weakening the
generic duplicate-Start block; focused tests passed `67/67` and full SQLite and
PostgreSQL suites passed `304/304`.

The exact S0 Start and same-session Stop were each published once,
acknowledged, reconciled, and marked replay-disabled. Two stable stopped-clean
observations supported S0 `CASE_PASS`; adjudication evidence `1975` credited
S0 observation 1 of 3. B's bounded proving window advanced state, heartbeat,
publish sequence, and tick monotonically. A remained healthy and running, the
UI/database/service state agree, and the temporary B firewall allowance was
removed. Evidence:
`work/evidence/watchdog-stage-7-controller-b-start-and-observation-20260809.md`.

## Current Context

Stages 0–7 are complete. Controller A is actively running the audit-selected
second S4 `44640 / AFTER` observation under the Pi-owned watchdog. Its Start is
acknowledged and replay-disabled; the supervisor, UI/database projection,
workers, evidence/backup/disk checks, and notification delivery agree and are
healthy. The exact CID remains `05000C61736466672200001C8901A34F` and the boot
witness remains `26B10B1280443248`.

Controller B specimen `F1-B1` completed and received credit for its first S0
observation under run `ffb59651-0bb0-4d9b-a729-0099cc0f5690`. Its one Start
and one same-session Stop are acknowledged and replay-disabled; the card is
stopped clean with unchanged CID and boot identity. The traveler reports
`HPT2_MEASUREMENT_PARTIAL`; any later B case requires separate admission and
authority.

The owner's requested B "preplay" continues to mean the complete Controller B
preflight/readiness process, not a bypass of the governed gate.

## Recommendation

Make the Pi application and its durable workers the sole campaign authority.
Add a persistent qualification watchdog that reconciles controller truth,
orchestration truth, and application/database truth. Prove its failure and
recovery behavior in fixtures and a production-safe live observation before it
is allowed to supervise recovery of A. Recover A without replaying Start, then
allow B preflight and Start only after a governed A/B parallelization gate is
implemented and passes fail-closed tests.

Do not use a larger fixed timeout as the primary correction. Do not keep Windows
as a hidden continuation dependency. Do not let the watchdog automatically
Start, Stop, reset, reflash, power-cycle, replace media, approve evidence, or
change a qualification outcome.

## Goal

Deliver an end-to-end qualification supervisor that detects and reports every
material campaign stoppage or state divergence, survives its own restart,
shows authoritative operator guidance, and safely supervises recovery of the
existing Controller A observation followed by a fully preflighted Controller B
Start.

## Non-negotiable Safety and Authority Boundaries

- No controller command is authorized by this planning artifact.
- Never replay an acknowledged or outcome-unknown Start or Stop operation.
- Controller state and exact run/card/firmware identity are authoritative over
  stale orchestration or database projections.
- The watchdog is detection, reconciliation, and notification infrastructure;
  state-changing actions require their own exact authorization envelope.
- SD cards may be removed, inserted, or replaced only with controller power
  off. Powered SD-card hot swapping is prohibited.
- Preserve append-only evidence and record invalid attempts separately from
  card failures.
- Controller A recovery and Controller B Start each require fresh exact-target
  authority after their immediately preceding gates pass.
- Any identity change, counter regression, unexplained error, ambiguous command
  outcome, or unavailable recovery evidence fails closed.

## Required Watchdog Contract

The watchdog samples and reconciles three planes:

1. **Controller truth:** controller role, firmware, card CID, boot witness, run
   ID, state, heartbeat, attempted/committed/verified counters, checkpoint,
   lane health, latched failure, and terminal proof.
2. **Orchestration truth:** expected stage and service, invocation identity,
   process and unit state, observation lease, last successful poll, progress,
   result, next transition, and supervisor heartbeat.
3. **Application truth:** campaign/run/stage state, command intents and
   acknowledgements, evidence finalization, interventions, notification outbox
   and delivery, operator action, deployment identity, and forecast validity.

It must create a durable, deduplicated intervention and prominent web status
when any of the following occurs:

- an expected service exits, fails, disappears, or does not start its dependent
  stage;
- an active database run has a stale heartbeat;
- a controller is progressing or held with no valid observer;
- an observer exists without the matching active controller/database run;
- run, firmware, card CID, boot, controller role, or counter identity diverges;
- progress stops beyond the profile's no-progress lease;
- a forecast is based on inactive or divergent execution;
- the watchdog, command worker, notification worker, or notification delivery
  becomes unhealthy;
- evidence storage, database backup, disk margin, or deployment-manifest checks
  fail;
- a campaign-owned remote process becomes stuck or orphaned.

The observation lease is progress-aware, not a single eight-hour wall clock.
It may renew only while exact identity is unchanged, counters and heartbeat are
monotonic, and no error is present. Every profile retains a separately derived
no-progress threshold and hard absolute review bound; renewal cannot continue
silently forever.

## Delivery Stages and Hard Gates

| Stage | Outcome | Entry Evidence | Exit Evidence / Hard Gate |
| --- | --- | --- | --- |
| 0. Preserve and baseline | Current A, B, database, services, processes, deployment, and notification state are captured without mutation | Owner authorizes execution of the plan; no target command is implied | Verified database backup and restore check; aggregate evidence manifest; exact read-only A/B snapshots; current service/process inventory; A remains held and B remains not started |
| 1. Implement the supervisor | One versioned Pi-owned watchdog, event contract, lease policy, UI projection, and notification path exist | Stage 0 passes; source and rollback baseline are clean and identified | Focused and full automated tests pass; systemd/Compose supervision uses restart/failure handling; watchdog has its own dead-man health; deployment manifest binds source, images, scripts, units, schema, and profiles |
| 2. Qualify failure detection | Normal, stopped, stale, divergent, restart, and notification-outage cases are proven without controllers | Stage 1 passes in fixture-only mode | All fault cases create the correct durable event and web state within two watchdog intervals; unexpected stops queue intervention email; provider acceptance is proven; normal planned transitions create no false intervention; notification failure is itself visible and retryable |
| 3. Deploy and prove production-safe monitoring | Exact reviewed supervisor release is live while state-changing adapters remain disabled | Stage 2 evidence reviewed; pre-deploy backup and rollback package pass | One compatible deployment manifest is live; health includes qualification-supervisor truth; watchdog restart/recovery passes; Windows is no longer an authority; campaign-owned stuck-process cleanup is controlled and evidenced; no controller command was issued |
| 4. Recover and resume A | Existing A observation is reconciled/finalized and the A chain resumes under supervision | Fresh exact A snapshot matches run, firmware, CID, boot, counters, checkpoint, and zero-failure baseline; Stage 3 remains healthy; fresh A recovery authority exists | Existing Start is not replayed; exactly one authorized Stop/finalization is reconciled or published as appropriate; the `44640 / AFTER` observation is credited or explicitly classified; next A stage starts once; watchdog, UI, database, and email state agree; A demonstrates stable supervised progress |
| 5. Enable B preflight policy | The A-before-B gate is replaced only by a reviewed evidence-based parallelization policy | Stage 4 passes and A remains stably supervised; resource-isolation analysis shows A and B can be monitored independently | ADR/policy states the minimum A evidence required before B; fail-closed application tests prohibit bypass, shared-resource collision, and Start while any B preflight blocker exists; reviewer-visible reason and audit event accompany the gate change |
| 6. Complete B preflight | B is ready for one exact controlled Start | Stage 5 passes; no unresolved campaign-wide watchdog incident exists | Every B readiness item below passes with immutable evidence; UI reports B ready and names the exact permitted action; no B Start intent exists yet |
| 7. Start and observe B | B begins once under the proven watchdog while A continues independently | Stage 6 passes; fresh exact B Start authority exists; final read-only identity snapshot still matches | One Start intent is prepared, published once, acknowledged, and reconciled; controller, database, service, and UI agree; watchdog observes monotonic B progress for the defined proving window; no A interference, notification failure, or resource collision occurs |

Failure of any stage returns the campaign to a visible safe hold or blocked
state. It never automatically advances to the next stage.

## Watchdog Qualification Matrix

At minimum, fixture and integration tests must cover:

- normal run, planned hold, planned Stop, clean terminal result, and dependent
  stage transition;
- observer failure while controller continues progressing;
- observer failure after controller reaches hold;
- stale database heartbeat with live controller, and stale controller with live
  database run;
- unknown Start/Stop publication outcome and acknowledgement reconciliation
  without replay;
- watchdog process termination, host reboot, database restart, command-worker
  restart, and notification-worker restart;
- controller communication loss followed by same-identity recovery and by
  changed-identity recovery rejection;
- no progress, slow progress, progress-lease renewal, hard-bound review, counter
  regression, and latched controller failure;
- email provider outage, delivery retry, deduplication, reminder/escalation,
  and visible failed-delivery state;
- expected versus unexpected service exit and prevention of false alarms during
  a normal stage transition;
- mixed or unknown deployment revisions, missing script/unit hash, stale
  forecast, low disk margin, missing current backup, and evidence-write failure;
- concurrent synthetic A/B runs proving independent locks, services, state,
  notifications, and evidence namespaces.

## Controller B Full Preflight

All items must pass immediately before preparing B's Start intent:

- [x] Controller role B, endpoint, probe, registry binding, and independent lock
      are exact and do not overlap A.
- [x] Exact firmware bundle, complete-readback hash, mailbox/profile contract,
      and current boot identity pass.
- [x] F1-B1 physical specimen assignment, visible identity, source, expendable
      status, custody, and prior-use record are complete.
- [x] Card installation/replacement occurred only with controller power off and
      is followed by a fresh complete service-467 CID identity block.
- [x] Protocol card status, CID validity, representative-sector read, decoded
      CID fields, and raw 128-bit CID pass and are linked to the run baseline.
- [x] Required initialization, scrub/baseline, lane, geometry, and zero-latched-
      failure checks pass for the exact B identity.
- [x] Campaign, assignment, profile, stage, limits, no-progress lease, hard
      review bound, evidence destination, and expected next transition are
      immutable and reviewer-visible.
- [x] Database backup is current and restore-verified; evidence storage and disk
      margins pass.
- [x] Command worker, notification worker, watchdog, email delivery, web banner,
      supervisor dead-man, and deployment manifest are healthy.
- [x] No B command intent or unknown prior outcome exists; any prior operation is
      terminal and reconciled.
- [x] A remains healthy under independent supervision, or is safely blocked in
      a state explicitly allowed by the parallelization policy.
- [x] The application computes `controller_b_execution_allowed=true` from the
      reviewed policy and evidence; no manual database override is used.
- [x] Fresh exact B Start authorization names controller, card, firmware,
      profile, run/stage, expiration, and single-use intent.
      Stage 7 envelope `5ef65727-8279-4cfa-88ca-979acc5393b2` and machine
      intent `209` supplied the single-use authority.

## A Recovery Decision

At Stage 4, compare a fresh read-only snapshot with the preserved baseline.

- If run ID, firmware, CID, boot witness, checkpoint, counters, lane health, and
  error state still match, reconcile the acknowledged Start, publish or
  reconcile one authorized Stop as required, finalize the held observation,
  and continue from the next uncredited matrix node.
- If the controller has already produced an equivalent terminal result,
  reconcile it; do not publish another Stop.
- If identity changed, counters regressed, errors appeared, state is ambiguous,
  or evidence cannot prove the existing run, classify the attempt blocked or
  inconclusive and stop for disposition. Do not replay Start to manufacture a
  cleaner history.

## Parallel A/B Policy Requirements

The current serial gate is not removed merely to save time. The replacement
policy must document which A results establish that firmware, profile,
orchestration, and evidence behavior are sufficiently proven for B to begin,
and which A closeout results remain independent card evidence that may run in
parallel. At minimum it must prove:

- the watchdog and command path have survived the A recovery and proving window;
- controller locks, probes/endpoints, service identities, evidence namespaces,
  command intents, and notification keys are role-specific;
- a failure or hold on either controller cannot advance, stop, reset, overwrite,
  or misattribute evidence to the other;
- shared Pi, database, storage, and notification capacity remain within measured
  margins with both synthetic roles active;
- B failure cannot be used to retroactively pass A, and A completion cannot
  automatically dispose B.

## Prospective Failure Controls

1. **False-green supervisor:** prevent with three-plane reconciliation and
   externally checked supervisor heartbeat; detect through injected observer,
   database, and watchdog failures before live use.
2. **Duplicate or misdirected controller command:** prevent with durable
   single-use intents, exact target identity, per-controller locks, and
   acknowledgement reconciliation; unknown outcomes always fail closed.
3. **Unsafe parallel A/B interference:** prevent with policy-backed independent
   resources and concurrency tests; any collision, capacity breach, or identity
   ambiguity blocks B before Start.

## Rough Sequence

1. Capture the current incident/baseline and create a current database backup.
2. Define the supervisor/event/lease contracts and move continuation authority
   wholly to the Pi.
3. Implement watchdog, UI, notification, manifest, and dead-man health.
4. Complete fixture fault matrix, backup/restore, exact release, and production-
   safe monitoring deployment.
5. Reconcile and resume A without replaying its Start.
6. Prove stable A supervision, then implement and review the parallelization
   gate.
7. Complete every B preflight item.
8. Obtain fresh B Start authority, start once, and verify supervised progress.

## Evidence and Reporting

Each stage produces a small versioned manifest recording timestamp, source and
deployment identities, commands or test IDs, target/controller/card identities
where applicable, outcome, hashes, and generated-evidence locations. The web
traveler must show the current gate, blocking reason, last verified controller
and orchestration observations, notification state, and the single safe next
action. Forecasts are suppressed whenever execution is inactive or divergent.

## Related Artifacts

- Project map: `work/maps/map-smi-firmware-stabilization-and-qualification.md`
- Parent application workpackage: `work/wp/active/wp-build-standalone-pi-sd-qualification-operator-application.md`
- Measurement checklist: `work/checklists/checklist-hpt2-bounded-stop-recovery-and-regression-measurement.md`
- Existing authorization decision: `work/decisions/decision-remaining-qualification-authorization.md`
- Endurance contract: `docs/hpt2-endurance-baseline-v1.md`
- SD operation/recovery truth: `docs/sd-card-operation-and-recovery.md`
- Endurance plan: `docs/sd-continuous-background-endurance-plan.md`

## Milestones

### Milestone 1: Watchdog completely qualified

- [x] Three-plane reconciliation, progress-aware lease, UI, notification, and
      dead-man health are implemented.
- [x] Complete fault matrix passes, including real email provider acceptance.
- [x] Exact production-safe deployment and rollback/restore proof pass.

### Milestone 2: Controller A recovered and progressing

- [x] Existing observation is reconciled without Start replay.
- [x] Held result is credited or explicitly dispositioned.
- [x] A resumes once and proves stable monitored progress.

### Milestone 3: Controller B preflight complete

- [x] Parallelization policy is reviewed, implemented, and fails closed.
- [x] Every Stage 6 B preflight checkbox passes with current exact identity evidence.
- [x] UI reports one exact B Start action and no unresolved blocker.

### Milestone 4: Controller B started under supervision

- [x] One authorized Start is acknowledged and reconciled.
- [x] B demonstrates monotonic progress during the bounded S0 proving window.
- [x] A and B remain independently supervised with no shared-state divergence.

## Open Questions Resolved During Execution

- Derive profile-specific no-progress and hard-review bounds from measured
  throughput; do not guess them in planning.
- Define the A proving-window duration from watchdog polling and observed stage
  transition timing before enabling B.
- Record the exact A evidence subset that permits B parallelization in an ADR;
  do not assume all A closeout is either required or irrelevant.

## Completion Standard

This workpackage is complete when the watchdog is demonstrably end-to-end and
restart-safe in production, the existing A observation is safely reconciled and
A is progressing under that supervisor, B's complete preflight and governed
parallelization gate pass, and B has exactly one acknowledged Start with
controller, orchestrator, database, UI, evidence, and notification state in
agreement.

## Closeout Routing

- Current truth promoted to: Pi `README.md`/`docs/`, applicable local SD
  operation and endurance docs, and the project map current checkpoint.
- Historical context remains in: this workpackage, the incident record, and
  stage evidence manifests.
- Runtime/status evidence: Pi campaign/evidence store plus compact manifests in
  `work/evidence/`.
- Cleanup deferred to: controlled retirement of obsolete Windows orchestration,
  ad-hoc host scripts, and campaign-owned orphan processes after replacement
  supervision is verified.
