# S4 parallel restart and watchdog run-boundary deployment — 2026-08-10

Status: PASS for runner recovery, parallel A/B continuation, and watchdog run-boundary verification; the ordinary S4 matrices remain active
Type: evidence
Updated: 2026-08-10
Next Action: preserve both active matrix runners; let B finish its third `10080 / BEFORE` observation and continue ordinary S4 while A finishes its third `44640 / AFTER` observation
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/checklists/checklist-hpt2-bounded-stop-recovery-and-regression-measurement.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Recovered evidence and root causes

Controller A run `de340af2-d505-495e-a14f-5159e55b1633` had already
completed and credited S4 `44640 / AFTER` observation 2 of 3. Its parent
service failed only after finalization because an A-only global Docker-process
probe treated concurrent Controller B tooling as an A failure. The measurement,
single Start, single Stop, stable stopped-clean observations, and both
adjudications remain valid.

The continuation audit also found three orchestration defects before replay or
new target access:

- the A observer remained bound to the completed run after its parent failed;
- the generalized observer used coarse stage `S4` instead of the exact
  `S4_<target>_<region>` identity, and its 5-second cadence could not make the
  protected latest-three-snapshot window span 60 seconds;
- the watchdog compared the first heartbeat and counters of a newly admitted
  run with the completed prior run for the same controller, producing a false
  `CONTROLLER_HEARTBEAT_REGRESSION` transition.

No prior Start or Stop was replayed. Before restart, A and B each had zero
active runs, zero prepared/published operations, inactive observers, and a
deny-only controller firewall.

## Unified correction and validation

The unified source commit is
`91bfd8a1709ea87aa9be393ac1a3bab086c77feb`. It contains:

- `51750c2`: per-role A/B observer lifecycle and role-scoped cleanup, with the
  invalid global cross-controller Docker probe removed;
- `1e6e7b6`: exact S4 boundary stage and transition identity;
- `f9632d1`: 30-second observer cadence, which remains inside the 90-second
  freshness bound and makes the latest three samples cover more than 60
  seconds;
- `91bfd8a`: monotonic watchdog state is carried only within the same run ID;
  a new run establishes a fresh heartbeat, progress-lease, and counter
  baseline while retaining current CID, firmware, boot, service, and manifest
  validation.

Validation passed:

- operations suite: `44/44`;
- focused watchdog suite: `28/28`;
- complete exact ARM64 application suite: `305/305` in `84.293 s` using a
  disposable writable evidence tmpfs;
- migration check: no model changes;
- shell syntax, Compose candidate rendering, Git diff check, deployment
  manifest validation, and immutable release-package checksums.

## Production release and rollback

Production release `qualification-watchdog-91bfd8a` uses image
`sha256:819e12a233a867d4fedd28c1f45c8247b4d1b27bcfe6f0792dcb265765fcf7e4`
and deployment manifest
`6bc6e768fb82728e8d1ec6af3a64be1f8ea8ac0c43c350c251c939ed3374cd4d`.
The immutable offline package is
`/home/jong/releases/qualification-watchdog-91bfd8a`.

Predeployment backup
`/backups/sd-qualifier-20260810T122513Z.dump` is `1233117` bytes with SHA-256
`2125e74c30d517a0fcc8e410d0b42a25898463c00557ae3f658e7edd617d9b60`;
its disposable PostgreSQL 17 restore passed at `2026-08-10T12:25:16Z`.
Rollback configuration and the prior image are retained under
`/home/jong/.local/share/sd-qualifier/rollback-watchdog-91bfd8a-20260810T1232Z`.

Migration and storage preflight passed. Web, command worker, notification
worker, and qualification watchdog were recreated on the exact new image; web
health is `healthy`. Both controller-runner services retained their original
run-service invocations. Both observers restarted, produced new-manifest
snapshots, and returned `HEALTHY / QUALIFICATION_SUPERVISOR_HEALTHY` without an
unresolved command. Committed progress advanced across the deployment from A
`2557 -> 2642` and B `9093 -> 9511`.

## Live A/B continuation proof

Controller A service `sdq-controller-a-s4-matrix-v4.service`, invocation
`496dbd96a55e45aaa6333b7fbd8f5a67`, admitted run
`a2dbfca4-805b-4fca-bf69-b01717e0f083`. Start
`f0d4724a-4e31-4cd0-a674-e19666f8cc81` was acknowledged once. The first
progress observation was `15/15/14`; after deployment A remained healthy and
advanced through `3213/3213/3212` attempted/committed/verified. Its parent
service invocation is still active, and role-A observer service
`sdq-qualification-observer.service` is active.

Controller B service `sdq-controller-b-s4-matrix-v4.service`, invocation
`fa196d1f1db54a6c8a4fd9c806f48c00`, completed run
`c46bc49d-c200-4dfe-a00d-ec38cfca7b2c` at exact `10079/10079/10079` for S4
`10080 / BEFORE`. Stop `dcda3faa-5a8b-4ad9-86af-25dd9cea0589` was
acknowledged once; both stopped-clean observations and adjudicators passed,
raising the row to observation 2 of 3.

The same parent then admitted run
`79a3a85f-3433-4d49-97a6-5bd539cadb41` for observation 3. Start
`0e28070b-77f8-4fce-8a02-c99064e82848` was acknowledged once. Its first
progress sample was `76/75/75`, and its first new-run supervisor snapshot was
immediately `HEALTHY` on the exact run, stage, service, release, and manifest.
It then advanced through `1451/1451/1450`; its parent invocation and
role-B observer invocation `aeaf67d2301f44c58329275d04c14ad3` remain active.
This is live target-environment proof that the prior cross-run heartbeat false
alarm is corrected.

At `2026-08-10T12:41:25Z`, both parent services and both role observers were
`active/running`. The application database projected both exact runs
`HEALTHY / QUALIFICATION_SUPERVISOR_HEALTHY`, with zero unresolved command
operations. The Controller B parallelization policy allowed independent
preflight and correctly blocked another Start because B already had an active
Start. The controller firewall was deny-only between bounded observer reads.

## Current card identities

Controller A same-session admission identity:

- protocol card status: detection state `5`; driver status `0`;
- CID valid: true; representative-sector read: true;
- manufacturer/listing and MID: unknown CID manufacturer, retail KEXIN,
  `0x05`;
- OEM/application ID: raw `0x000C` (decoded `??`);
- product/revision: `asdfg` / `2.2`;
- serial/manufacturing date: `7305` / `2026-03`;
- complete raw 128-bit CID: `05000C61736466672200001C8901A34F`;
- boot-session witness: `26B10B1280443248`.

Controller B same-session admission identity:

- protocol card status: `READY; RAW_HPT; DRIVER_OK`;
- CID valid: true; representative-sector read: true;
- manufacturer/listing and MID: unknown CID manufacturer, retail KEXIN,
  `0x05`;
- OEM/application ID: raw `0x000C`;
- product/revision: `asdfg` / `2.2`;
- serial/manufacturing date: `7298` / `2026-03`;
- complete raw 128-bit CID: `05000C61736466672200001C8201A379`;
- boot-session witness: `F54F10127F443248`.

No SD card was removed, inserted, reinserted, or replaced, and neither
controller rebooted during this continuation. CID is identity evidence, not
proof of authenticity. Firewall allowances remained controller-, port-, and
owner-scoped during active observations and were removed by their owning
operations; the default drop rule remained in force.

## Reconciliation of the delivered stage-identity intervention

The `RUN_BLOCKED` email for "Application and orchestration stage identities do
not agree" is historical transition evidence, not the current supervisor
state. It was created as notification `176` from event `1073` at
`2026-08-10T12:00:19Z`, before the exact-stage observer correction and before
the unified production deployment.

The alerting snapshot
`74423efa62b923b8430d73d61089114be59035e0bdc0140de8fa3a533173bef5`
was bound to Controller A run `a2dbfca4-805b-4fca-bf69-b01717e0f083` and
showed application stage `S4_44640_AFTER`, orchestration stage `S4`, and
progress `9/9/8`. Event `1075` at `2026-08-10T12:03:52Z` records recovery on
snapshot `a8f5ffe93ec7deb0ee17680d0946c098168d2b0c550032fc00c0f6f9894303`,
where both stage identities were `S4_44640_AFTER` and progress had advanced to
`288/288/287`. No Start or Stop was replayed or inferred.

The email rendered "Controller role: not assigned" because watchdog
interventions are stored as campaign-level notifications with no `run` foreign
key. The durable transition, snapshot, and notification deduplication key all
identify role A. This is a notification-attribution limitation, not evidence
that the monitored controller was unknown.

Read-only reconciliation at `2026-08-10T12:58:18Z` found A healthy with exact
matching stage identities at `4544/4544/4543`, and B healthy with exact
matching stage identities at `7293/7293/7292`. Both parent runners and both
observers remained active, and there were zero unresolved command operations.
The delivered alert remains an append-only historical operator notification;
it was not silently acknowledged or closed during read-only reconciliation.

## Remaining gate

This repair does not complete the qualification campaign. A remains on the
third S4 `44640 / AFTER` observation, B remains in ordinary S4, and final
`129600 / AFTER`, identity-local S5/S6, recovery, representative A/B
regression, final scrubs/CID captures, combined review, and production Stop
deadline approval remain open.
