# Progress-aware runner supervision and Controller B S0 observation 3 — 2026-08-09

Status: PASS; A/B independent supervision is proved, Controller B S0 is 3/3, and the serialized B base runner is active
Type: evidence
Updated: 2026-08-09
Next Action: preserve the active A and B runners; let B finish S1/S2 base coverage under the serialized observer, then continue the remaining matrix
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/checklists/checklist-hpt2-bounded-stop-recovery-and-regression-measurement.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Corrective release

The fixed eight-hour matrix wait and observer-only orchestration projection were
replaced by a record-progress lease, a separate seven-day hard ceiling, and an
ordered expected-service chain. The service selector stops at the first active,
failed, or missing campaign runner, so an active observer can no longer hide a
failed matrix or dependent service.

Production initially ran release `qualification-runner-supervision-ffbf93e`.
After a live B observer/command collision exposed uncoordinated J-Link access,
production first advanced to release `qualification-runner-supervision-f664bf5`.
The final corrective deployment is
`qualification-runner-supervision-e82398a`, source commit
`e82398a0ce3a3bdd3039b39a1033ef1570b1b2ca`, unchanged application image
digest `sha256:f53c01017249e37afa93d06012ec3d9704159fcd72da66a8fbd30f03594a78d0`,
and deployment manifest
`a25039e4a9d3f364e8e08cd5210d874a14b98ba3266b42954b516074125fb1fc`.
The manifest binds the immutable supervisor source, progress evaluator,
campaign runners, full runtime-tree hash, latest restored database backup, and
policy profiles. This final release also removes the old Controller-B-at-INTAKE
assertion from every later A post-S4 and recovery script.

The new host lock serializes every target-bearing campaign Compose call with
the same controller's observer capture. A and B use separate lock files, lock
wait is bounded to 120 seconds, and process exit releases the descriptor.
Focused runner/service tests pass `20/20`, including a real two-process
same-controller serialization test. Live B Start and Stop publication then
completed while the B observer remained active, proving the collision was
removed. The production correction also traps forced observer exits and removes
same-role stale observer firewall allowances before each new target cycle.
Operations commit `9c00886` adds bounded idempotent adjudication confirmation,
binds the base matrix to the case runner from the same immutable operations
tree, and guards exact admitted/prepared-run recovery.

The final manifest also binds two independent systemd units. A uses
`qualification-supervisor.env`; B uses
`qualification-supervisor-b.env`. Both invoke the same stable, hashed observer
launcher. The B unit is static and inactive at rest, so it cannot start without
an exact admitted B run environment and cannot replace A's projection.
Rollback copies are retained under the exact timestamped
`/home/jong/.local/share/sd-qualifier/deploy-backup-*` directories.

Validation passed:

- focused runner/service-selection tests: `14/14` after the separately
  versioned B S0 continuation runner was added;
- complete exact-image application suite: `304/304` on disposable SQLite;
- prior same-source PostgreSQL 17 suite: `304/304`;
- shell syntax and `git diff --check`;
- fresh logical backup and disposable restore before deployment;
- web health, all three workers, deployment identity, and protected manifest
  verification after serial service recreation.

The superseding deployment backup is
`/backups/sd-qualifier-20260810T020831Z.dump`, `699464` bytes, SHA-256
`e94e732a2ee9f74fe8552e41a1ee1ad89a046c460bef9d26f81909f4df8698b9`;
its disposable PostgreSQL 17 restore passed before service recreation.

The corrected runtime uses both the protected production environment and the
supervisor environment with the immutable Compose file and override. This
eliminates the stale mutable-repository identity that initially prevented
short-lived target adapters from reading the new protected manifest.

## Controller A recovery result

Controller A resumed exact run
`de340af2-d505-495e-a14f-5159e55b1633`; its sole Start remained
`ACKNOWLEDGED` and no new Start or Stop was created. The matrix runner invocation
is `8bf2ff84f9cb432db4b74edffe00ee6c`, and all three dependent continuation
services are active and waiting in order.

Three new-manifest proving snapshots span more than 60 seconds. The live web
projection is `HEALTHY / QUALIFICATION_SUPERVISOR_HEALTHY`, forecast-valid,
and names `sdq-controller-a-s4-matrix-v3.service` rather than the observer.
After Controller B observation 3, A remained on the same identity and advanced
to attempted/committed/verified `22321/22321/22320`:

- role: `A`;
- firmware: `smi-00-05-00-660433743898-bcf70c2de65c-release`;
- boot witness: `26B10B1280443248`;
- complete raw CID: `05000C61736466672200001C8901A34F`.

The focused service-selector fixture proves a failed runner is selected ahead
of an active successor. The complete application suite covers failed-service
projection to `ORCHESTRATOR_STOPPED`, deduplicated intervention creation, web
visibility, and retryable email delivery failure.

## Controller B S0 observation 3

The protected parallel-entry policy returned
`CONTROLLER_B_PREFLIGHT_ALLOWED_START_BLOCKED` before admission, with only the
expected not-ready-stage and no-active-run blockers. A fresh database backup
`/backups/sd-qualifier-20260810T010323Z.dump` is `624233` bytes, SHA-256
`4e1d0a1b7069b3bc56f3aae42658fda4f7e01d36c05eb06c5e372b0137e4c45f`,
and passed a disposable PostgreSQL 17 restore.

Immutable operations commit `021773d6248a1ee59ed8a545a6de51ff58d538d0`
admitted and completed exact B run
`c20498c3-9ba6-476d-b240-d868b7937044`. Only B port `19121` was temporarily
allowed, and its exact rule was removed by the exit trap. At rest, the default
drop rule again covers `{19120,19121}` and only the persistent A observer
allowance remains.

Controller B identity evidence `2087`, SHA-256
`71212601dbc1918d3a3eecc329ffdd656d495638933e5f816b5c9d55f051116a`,
records:

- protocol card status: `READY; RAW_HPT; DRIVER_OK`;
- CID valid: true;
- representative-sector read: true;
- manufacturer: unknown CID manufacturer; retail brand KEXIN; MID `0x05`;
- OEM/application ID: raw `0x000C`;
- product name: `asdfg`;
- product revision: `2.2`;
- serial number: `7298`;
- manufacturing date: `2026-03`;
- boot witness: `F54F10127F443248`;
- complete raw 128-bit CID: `05000C61736466672200001C8201A379`.

The card was not removed, reinserted, initialized, or rebooted after the prior
same-card capture. No powered SD-card insertion or removal occurred. CID is
identity evidence, not proof of authenticity.

Start `d99104da-d7ed-4ff4-a8b4-646d8f2362d8` and Stop
`e9ae9e12-2700-49f9-ab1c-0393ee546b74` are both `ACKNOWLEDGED` with terminal
code `MAILBOX_COMMAND_ACKNOWLEDGED`; no unresolved B operation remains.
Same-session evidence `2096` has SHA-256
`e9abdd8b984df6d064aae891d5592650f853e867d48ff615bc14a24fc804e477`
and records `0 ms` from Start acknowledgement to Stop publication. Two
stopped-clean observations retain state `5`, terminal reason `0`, terminal
proof `31`, the same boot witness, and the same CID.

Measurement evidence `2099`, SHA-256
`9aa89f4f359e2aa51cdaad030dc6d5de46096ad4805d5780fb0fd4f07c79f933`,
is `CASE_PASS` with bounded Stop and a `2170 ms` Stop-to-terminal observation
upper bound. Adjudication evidence `2100`, SHA-256
`439350820b1e95fb4eeb19ac051c6ffc26454ab488386d4779c11664f018a6ea`,
credits S0 and S3. Controller B S0 is now `3/3`, PASS.

## Remaining gate

The independent B supervisor was proved on S1 run
`fdf1d523-1a36-402d-b45d-257e6e58ab38` using runner invocation
`051cf153402547f68cb725f9a73f6761` and observer invocation
`6e8adf191c774f57aa2439e985353b08`. B projected `HEALTHY` with exact CID,
boot, firmware, run, service, and monotonic `380/380/380` progress while A
independently remained `HEALTHY` at `23839/23839/23838`. The run completed
bounded Stop with both operations acknowledged and no unresolved operation or
firewall allowance. Its first zero-delay S1 attempt did not meet the S1 trigger
and was correctly not credited; this is measurement evidence, not supervisor
failure.

The successor B authorization envelope is
`7da70559-4020-4d46-bf67-053a4c0eede8`, with a hard review at
`2026-08-16T00:00:00-07:00`. The ambiguous B Start
`795076e1-43f6-4e8f-9f8c-80b9e7876e7b` was never replayed: a paused-observer
read-only audit proved its exact mailbox, acknowledgement, request/run, boot,
and CID binding; official reconciliation then acknowledged it, and paired Stop
`c91d5f2e-569e-4e2e-ba3f-ad8358d87dee` completed the run. Result evidence
`2299` (`a4bb4371...36cb`) and adjudication `2301`
(`ac69afd1...4d13`) credit one S1 observation.

The next clean bounded case, run
`c12a1067-b5dd-438b-bc72-dc56df10386c`, credited S2 after a transient first
adjudication invocation was safely confirmed by rerun as evidence `2327`.
The corrected immutable B parent invocation is
`9e0a35802e604904b783ce22ebf49fc6`; its first complete production attempt
passed end to end and raised B base coverage to S1 `1/10`, S2 `2/10`, S3
`3/10`. Controller A remained on original runner invocation
`8bf2ff84f9cb432db4b74edffe00ee6c` and advanced to
`26716/26716/26715` with no Stop or Start replay.

The A and B matrices remain in progress. B still requires the balance of
S1–S6 and S4-region coverage. Do not admit the post-matrix
recovery/regression campaign until the combined matrix audit passes.

After the final deployment, a Controller B admission completed target reads
but its parent was stopped before creating a command. Exact run
`a773727d-d757-4a51-bf03-18eef733207f` was proved `RUNNING` with zero
operations. A transient database preparation failure left that state unchanged;
one database-only retry created exact prepared Start
`d3a4d0e1-e3ed-4123-b77c-c957bafec01c`. Guarded operations commit `9c00886`
published that intent once, paired it with acknowledged Stop
`36c418ea-7445-4c32-a694-1f70f19031f6`, and completed the run without replay.
Result evidence `2470` has SHA-256
`12de470f9d543d66c8bf01fa1b53b7b845decf561b1e82bda78c8cdefac26464`;
adjudication evidence `2473` has SHA-256
`a02964855ae767c5fd281f079d01fef54ce5969b0c4b3e79e5e66cf070ad17fe`.

At restart, durable B coverage was S0 `3/3`, S1 `2/10`, S2 `6/10`, and S3
`3/10`, with zero unresolved B commands. New parent service
`sdq-controller-b-base-matrix-v3.service`, invocation
`8bd7f82b0afe4c63ab33115e4557d532`, admitted the next S1 case and launched
the independent B observer. Controller A remained active on original runner
invocation `8bf2ff84f9cb432db4b74edffe00ee6c`; no Start or Stop was replayed on A.
