# Repair progress-aware measurement-runner supervision Workpackage

Status: completed
Type: workpackage
Updated: 2026-09-23
Next Action: none; retained as history; any new execution requires a new explicit owner instruction
Campaign Reason: Historical record from campaign 009-complete-clean-contract-bound-a-b-qualification; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md
Evidence: work/evidence/after-measurement-matrix-ship-readiness-gate-audit-20260809.md
Completion Evidence: work/evidence/progress-aware-runner-supervision-and-controller-b-s0-3-20260809.md

## Problem

The production watchdog reports Controller A healthy because its orchestration
snapshot names the observer service. The actual S4 matrix runner and all three
dependent continuation services are failed after a fixed eight-hour wait
expired during healthy forward progress. The controller run remains valid and
continues, but no durable runner is present to observe the exact boundary and
advance the chain.

This violates the reviewed watchdog contract for expected-service exit,
dependent-stage transition, progress-aware leases, and authoritative operator
guidance. It is an infrastructure failure, not a firmware or card failure.

## Goal

Make orchestration truth include the exact expected campaign runner and next
transition, use a progress-aware observation lease with a separate hard review
bound, and prove safe recovery of the current acknowledged run without replay.

## Requirements

- Preserve exact run, controller, firmware, CID, boot, checkpoint, and command
  identity. Never replay the acknowledged Start.
- Project the actual matrix runner and expected dependent service, not merely
  the always-on observer.
- Renew the observation lease only while identity is unchanged, heartbeat and
  counters are monotonic, and no error is present.
- Retain a separately reviewed hard absolute bound and visible intervention;
  no fixed wall limit may discard healthy progress silently.
- A runner failure or missing transition must produce web and email
  intervention and suppress completion forecasts.
- Resume/finalize only through existing no-replay operation reconciliation and
  exact authorization envelopes.
- Keep Controller B independently firewalled and inactive unless a fresh exact
  B operation is separately authorized.

## Failure controls

1. Wrong-run or duplicate-command recovery is prevented by exact identity,
   existing-operation count/state assertions, and read-only reconciliation.
2. False-green supervision is detected by fixture cases for runner exit,
   missing dependent transition, observer-only health, and stale projection.
3. Unbounded silent execution is prevented by progress-only lease renewal,
   explicit hard review, durable intervention, and notification verification.

## Exit gate

- [x] Focused runner/watchdog tests and the complete Pi application suite pass
      on disposable SQLite and PostgreSQL 17.
- [x] Deployment manifest binds the corrected source, image, observer, runner,
      and unit identities; backup/restore and rollback checks pass.
- [x] Production shows the exact A runner active and the watchdog names that
      runner and its expected next transition.
- [x] The existing acknowledged A run is resumed without Start replay and
      demonstrates healthy progress through at least three new snapshots.
- [x] A forced fixture runner exit creates one deduplicated web/email
      intervention and cannot project healthy.
- [x] No unapproved B command, card movement, or powered media manipulation
      controller operation occurs.

## Completion standard

Complete when the production supervisor cannot be green while the required
matrix runner or dependent transition is failed, and the current A chain is
safely progressing under the corrected, restart-safe, progress-aware contract.

## Completion

Completed by production release `qualification-runner-supervision-ffbf93e`.
The A runner and all dependent units are active; the supervisor names the
runner invocation and has three new-manifest proving snapshots. The existing A
Start was not replayed. The separately authorized B S0 observation 3 completed
without card movement and does not weaken this corrective gate.
The manifest-bound independent B observer subsequently passed a live S1
attempt while A remained independently healthy; A and B retain separate
environment files, service identities, snapshots, and notifications.

## Owner-directed reconciliation — 2026-09-23

Reconcile stale active lifecycle to the previously documented completed disposition; no new test or broader acceptance claim.

Authorization: Owner instruction in this conversation on 2026-09-23 to close/abandon the named work and mitigate older active records.

Previous header retained for evidence interpretation:

> Status: complete; Controller A is progressing under runner-aware supervision and the continuation chain is active
> Next Action: preserve the repaired A chain and qualify an independent B supervisor before admitting longer B cases

Earlier observations, unchecked historical criteria, and procedures below/above remain history. This administrative disposition does not qualify a new firmware candidate, physical card, thermal system, ECO, or release. Current 00.05.01 obligations in IDEA-0006, IDEA-0007, MAP-0009, and CAMP-0028 remain unchanged.
