# Watchdog Stage 4 Controller A recovery and resume evidence — 2026-08-09

Status: PASS; held A run finalized without replay and the next audited A run is making stable supervised progress
Type: evidence
Updated: 2026-08-09
Next Action: begin Stage 5 policy work only; do not preflight or start Controller B yet
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/wp/completed/wp-watchdog-gated-controller-a-recovery-and-controller-b-start.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Fresh recovery identity gate

Read-only operation `dfc4dbee-38a0-4902-b7e6-8a118d64b495` completed PASS at
`2026-08-09T20:29:03Z` against Controller A, assembly `1751136`, probe
`69664082`, hardware variant `0`, and firmware bundle
`smi-00-05-00-660433743898-bcf70c2de65c-release`.

The observed full-flash SHA-256 matched expected
`507e5b91d53afeb1b2d9d16442e3f6ff0391dcc7a53cead269cb11396dfcf246`.
The controller was at the exact preserved held boundary: run
`717ae30c-736d-430c-8ce5-529c97b3920a`, checkpoint `4`, counters
`44641/44641/44641`, boot witness `26B10B1280443248`, three healthy lanes,
zero system/HPT/driver errors, zero mismatch/loss, no media replacement, and no
retired segments.

### CID identity block

- Protocol card status: detection state `5`; driver status `0`.
- CID valid: `true`.
- Representative-sector read: `true`.
- Manufacturer/listing: KEXIN retail specimen; MID `0x05`.
- OEM/application ID: `??`.
- Product name: `asdfg`.
- Product revision: `2.2`.
- Serial number: `7305`.
- Manufacturing date: `2026-03`.
- Raw 128-bit CID: `05000C61736466672200001C8901A34F`.

No SD card was removed, inserted, or replaced. No powered media manipulation
occurred.

## Held-run stop, finalization, and adjudication

The prior acknowledged Start
`dbba74bc-d652-4eab-90f3-7735b0708afe` was never replayed. The existing active
authorization envelope `3f8f2f98-a2dc-4245-be6f-e0318f6fc0b9` bound A, the
exact CID, assembly/probe, firmware manifest, and Start/Stop actions.

Exactly one Stop operation was prepared and published:

- Operation: `47d3dff1-7f07-4dcd-876e-373dd059aeea`.
- Request: `fa3e42a8-eb7f-43be-9a01-86b27bb96451`.
- Command/publish sequences: `206` / `412`.
- Result: `ACKNOWLEDGED`, `MAILBOX_COMMAND_ACKNOWLEDGED`.
- Status SHA-256:
  `c5543445cb6d519c35092357d12b63316985bd370c750f14059f9f5fe1304194`.
- Automatic replay allowed: `false`.

Four consecutive observations, Evidence `1623`–`1626`, proved state `5`,
terminal proof `31`, reason `0`, the same CID/boot identity, and stable clean
terminal fields. Database finalization Evidence `1629` passed bounded Stop with
a `2062 ms` observation upper bound. Base adjudication Evidence `1632`
correctly classified the exact pre-Stop counters as S3-shaped rather than
granting the planned label. Extended adjudication Evidence `1633` preserved
the admitted S4 `44640 / AFTER` checkpoint observation. The matrix audit then
reported A `44640 / AFTER` at `1/3`; this explicit audit selected the next
node.

Two finalization/adjudication attempts encountered PostgreSQL deadlock victims
against the continuously updating supervisor. Each aborted transaction was
checked before retry; no duplicate evidence or controller action resulted.

## Next A run and one-shot Start

Fresh read-only admission created run
`de340af2-d505-495e-a14f-5159e55b1633` for S4 `44640 / AFTER` replicate 2.
Admission Evidence `1638`/`1639` binds the same CID, boot, bundle, target, and
region. Before Start, the active envelope and zero-operation state were
rechecked.

Exactly one Start was prepared and published:

- Operation: `ec86b221-43bb-4fa4-9d2f-f5ea6f838c45`.
- Request: `c34d0b93-a829-4d24-abfc-60b0b407e634`.
- Command/publish sequences: `207` / `414`.
- Result: `ACKNOWLEDGED`, `MAILBOX_COMMAND_ACKNOWLEDGED`.
- Automatic replay allowed: `false`.

Controller B remained at `INTAKE`; it was not observed, preflighted, admitted,
or started during Stage 4.

## Live watchdog correction and release

The first live progress snapshots exposed a false positive: firmware
intentionally clears `LastReadLaneMask` while a read is in flight, so the
coherent `attempted == committed == verified + 1`, append-mask `7`, read-mask
`0` state was incorrectly labeled `CONTROLLER_FAILURE_LATCHED` despite
`latched_failure=false` and continuing progress.

Commit `6e26d778852ae24094af5939cb84936c639470f2` accepts only the firmware's
coherent reset/append/read transition windows. It still rejects partial lane
masks, zero masks outside a transition, incoherent counters, driver/HPT
errors, and true latched failures. Focused tests passed `31/31`; the complete
image suite passed `295/295` using a disposable writable evidence root.

Current production identity:

- Release: `/home/jong/dev/sd_qualification-release-6e26d77`.
- Image:
  `sha256:14094d1659c54a9ccf9738022b5d84498e1ae0a021bec0fb16652042580e40ad`.
- Deployment manifest:
  `cbf5b303a02457bacd286ca3c52fbbcfa4fce862516781fbf62bc7a7aa091924`.
- Host observer unit:
  `1a05a6261f43c748b8169c15f7c6242a846d3bf963b56163f0b3b2a3ab8b7a6a`.
- Compose override:
  `76c20c76cf679690a763647ca6939d456495233d3c53e76494ab9934548c534a`.

Two deployment attempts failed closed before replacement: the first validated
a not-yet-installed override path; the second encountered one stale Docker
`Created` container name. The old production services and manifest remained
live. The exact non-running temporary container was removed, and the third
hash-gated attempt applied no migration, recreated the four application
services, reached healthy state, and restarted the observer with
`NRestarts=0`. No controller command was issued by any deployment attempt.

## Stage 4 exit proof

New-manifest snapshots `44`, `45`, `46`, and later live cycles remained
`QUALIFICATION_SUPERVISOR_HEALTHY`. Verified counters advanced monotonically
`1070 -> 1124 -> 1179 -> 1232 -> 1446`; controller heartbeat advanced; CID, boot,
run, role, firmware, and checkpoint stayed exact; `latched_failure=false`;
worker, storage, backup, disk, notification, and service checks remained
healthy. Web, command worker, notification worker, and watchdog all run the
exact new image, and the host observer is active/running.

Transient bootstrap and false-positive interventions were delivered by email
and remain append-only historical incidents. Current notification delivery is
`READY`; the current supervisor projection is `HEALTHY`.

Stages 0–4 are complete. Stage 5 is the next gate and must review and implement
the A/B parallelization policy. This evidence does not authorize Controller B
preflight or Start.
