# Recover and complete the Aug 10 SD qualification campaign Workpackage

Status: complete
Type: workpackage
Updated: 2026-08-13
Next Action: none

Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Current Context

### 2026-08-13 accepted closeout

The owner accepted the audit recommendation. The deployed application closed
campaign `8c7a2b1a-7e67-48ac-8895-7999d76c4c2a` without a qualification
conclusion, preserving `361` terminal HPT2 operations and both card results
unchanged. The application stores the operator-facing `INCONCLUSIVE` decision
as terminal `ABORTED`, version `3`; event `1691` records
`qualification_conclusion=false`, and event `1692` publishes the reconciled
manifest/report. Dashboard, detail, coordinator, ledger, and report now agree.
The exact campaign-bound host observer is disabled and inactive, and the
appliance is healthy under the documented between-campaign worker override.
Evidence:
`work/evidence/sd-qualifier-aug10-campaign-inconclusive-closeout-20260813.md`.

### 2026-08-13 frozen snapshot audit

A restore-verified logical backup and repeatable-read audit of campaign
`8c7a2b1a-7e67-48ac-8895-7999d76c4c2a` is complete. All `13,043` frozen
ledger rows matched their files, all `187` runs and `361` HPT2 command
operations were terminal, event sequence `1..1690` was contiguous, and the A/B
CIDs were stable across `105 / 85` captures. The audit admitted `136` effective
S0-S3 credit rows and retained `37` no-credit rows without reinterpretation.

The campaign should not resume: it is still `RUNNING` over a paused legacy
six-node plan, the selected `SMI-SDQ-CONTRACT-V1` is not bound, its valid
firmware bundle records `admitted=false / scope=none`, three coordinator tasks
are failed, supervisor projections are not current and agreeing, Controller B
S2 is only `2 / 10`, and S4-S6 remain incomplete. Evidence and recommendation:
`work/evidence/sd-qualifier-aug10-campaign-snapshot-audit-and-disposition-recommendation-20260813.md`.

The recommendation was accepted and implemented. A successor campaign was not
created because Gate 2 must first deliver the reviewed contract-bound release
baseline and exact new observer binding.

### 2026-08-13 contract-first pause boundary

The governing front door is now `docs/sd-card-qualification-contract.md`, and
the application source contains validated but not deployed or live-plan-bound
contract `SMI-SDQ-CONTRACT-V1`. The live campaign is a `CHANGE_IMPACT`
campaign; it cannot independently approve the KEXIN card family.

The coordinator is `PAUSED` with no claimed or pending task. Controller B S2
run `efdeaec7-f0e7-4458-8012-7580dab9f18f` is now closed. Its Start was
reconciled read-only as acknowledged, and exact Stop operation
`f1492992-f957-4baa-a676-471b8c2f995f` was published once and acknowledged.
Terminal observations `19267` and `19268` supported the non-promoting
finalizer. Result `19269` records
`BOUNDED_STOP_PASS_CASE_TRIGGER_NOT_MET`, `case_accepted=false`, and no S2
credit. Both role supervisor states are healthy and command count remains
`361`. Do not replay Start or Stop. The next step is the explicit campaign
migration-versus-INCONCLUSIVE decision before any execution resumes.

### Earlier recovery context

The live Aug. 10 HPT2 campaign is blocked at Controller B S1. Its controller-side
run completed all 129600 sectors with valid terminal proof, but the application
left the run `RUNNING` because the base-case runner only knows how to finish a
requested STOP. Its later STOP was correctly rejected by already-complete
firmware, and the campaign coordinator exhausted its bounded task attempt.

The deployed web closeout is commit `520014c`; the deployed coordinator source
is commit `8eaa33e`. They share base `928d2ce` and must be integrated before a
replacement release. No powered SD-card operation is permitted. Firmware and
physical-media changes are outside this authorized detour.

## Recommendation

Add a fail-closed application path that terminalizes an identity-matched base
run when authoritative supervisor evidence proves natural completion, including
the known rejected-STOP-after-completion case. Make coordinator cleanup consume
that result so bounded attempts cannot leave orphan `RUNNING` rows. Inspect S5
and S6 producer routing before changing it; an empty evidence matrix is not by
itself proof that producer code is absent.

Do not replay Start, manufacture matrix credit, edit campaign records directly,
or alter firmware/controller state to make the application agree with reality.

## Current State

Completed:

- Reconciled the original Start operation read-only as acknowledged.
- Captured current Controller B S1 identity and terminal supervisor evidence.
- Attempted the supported resume path once; it issued no Start replay and
  preserved the rejected STOP as durable evidence.
- Identified the exact deployed source split and common Git base.
- Integrated and deployed the web/coordinator source lines as release
  `campaign-completion-d4df0f2`.
- Recovered the orphan Controller B S1 run as clean natural completion with no
  case credit and durable evidence `16959`.
- Reconciled and resumed the failed coordinator task through the website.
- Proved fresh Controller A and Controller B S1 admission and execution through
  the corrected persistent coordinator.
- Deployed release `campaign-completion-7e836bf`, including atomic final Stop
  publication, bounded-attempt termination, and reviewed expired-lease web
  reconciliation.
- Passed `94 / 94` focused tests from the exact ARM64 image and a predeployment
  database backup/restore check.
- Reconciled the expired task through the web with all controller actions
  terminal, resumed exact attempt 4, and proved the first corrected live S1
  credit on Controller B run `282ef6dd-47ef-498f-8332-1199d7fcec23`.
- Added fail-closed pre-admission cleanup in `61e77a7`; attempts 4 and 5 ended
  with zero active runs and zero unresolved commands.
- Restored the stale Controller B J-Link read-only support path by restarting
  only probe service `69660123` and verifying a fresh read-only capture; no
  firmware, controller-power, or media operation was performed.
- Completed Controller B S1 at `10 / 10` on attempt 6 with genuine observed
  results and no inferred credit.
- Committed and installed `831e65c`, which releases an already-prepared
  Controller A Stop after useful B work ends instead of waiting out the full
  25-minute proving ceiling. Runtime/source hashes and shell syntax agree.
- Finalized the acknowledged-stop Controller B S2 run from two stable
  read-only terminal observations without S2 credit or command replay; both
  role supervisor projections are healthy.

Incomplete for the wider project, but outside this recovery workpackage:

- Build the reviewed contract-bound release baseline and conduct any remaining
  change-impact work in a clean versioned campaign.

## Goal

Complete the Aug. 10 campaign through the website-driven coordinator with every
run durably terminal, truthful progress/status, and an evidence-backed final
qualification conclusion or a genuine external gate stated explicitly.

## Why It Matters

The operator cannot trust or complete a campaign while authoritative controller
completion is represented as an active database run. Correct lifecycle state is
also required before the coordinator can safely continue the remaining matrix.

## Major Outcomes

- Integrate the web closeout and latest coordinator behavior in one release.
- Recover naturally completed base runs without command replay or invented pass credit.
- Prevent bounded coordinator attempts from leaving active run records.
- Validate and deploy the application change on `srv-rpi00.local`.
- Preserve the live campaign as terminal history and route remaining work to a clean campaign.

## Delivery Stages

Use stages when sequencing, parity gates, or handoff cost matter.

| Stage | Outcome | Entry Evidence | Exit Evidence |
| --- | --- | --- | --- |
| 1 | Correct run lifecycle | Terminal controller evidence and durable command history | Run closes with an explicit no-trigger disposition and no matrix credit |
| 2 | Correct coordinator recovery | No active controller command and terminal run state | Failed node can be safely reconciled/retried without an orphan run |
| 3 | Integrated release | Web and coordinator histories combined | Focused tests pass on ARM64 and target services run one verified release |
| 4 | Campaign disposition | Frozen audit and owner decision | Campaign closes without conclusion, controller replay, or card rejection |

## Related Artifacts

- Tickets: work/tickets/ticket-add-audited-campaign-closeout-to-sd-qualifier.md
- Checklists:
- Spikes:
- ADRs:
- Runbooks:
- Inventories:
- Decision matrices:

## Rough Sequence

1. Integrate the two deployed source lines in a clean branch.
2. Implement and test natural-completion disposition and coordinator cleanup.
3. Build and deploy the integrated app release.
4. Recover the current B S1 run and freeze the final audit.
5. Record the owner disposition, publish the report, and reconcile status.

## Milestones

### Milestone 1: Application detour complete

Completion criteria:

- [x] Current orphan run is terminalized from authoritative evidence without replay.
- [x] No clean no-trigger attempt earns a matrix pass.
- [x] Coordinator can continue after the orphan run is durably reconciled.
- [x] Focused and adjacent tests pass on the target architecture.
- [x] Integrated release is deployed and target services are healthy.

### Milestone 2: Aug. 10 campaign complete

Completion criteria:

- [x] Every existing run and HPT2 operation is terminal, with incomplete
  matrix/post-campaign work preserved as no-credit or not-run history.
- [x] Campaign has a terminal no-qualification-conclusion disposition supported
  by the frozen audit and owner decision.
- [x] Dashboard, campaign detail, immutable report, run records, and coordinator
  state agree.

## Resolved routing

- S5/S6 producers exist; any required S5/S6 execution belongs to a clean
  contract-bound successor campaign.
- Remaining qualification, firmware, and physical-media dependencies are not
  grounds to reopen or resume this historical campaign.

## Completion Standard

This workpackage is complete because the live Aug. 10 campaign has a truthful
terminal no-qualification-conclusion disposition, all applicable application
verification is recorded, and no active run, operation, task, observer, or
coordinator state contradicts that disposition.

## Closeout Routing

- Current truth promoted to: SD Qualifier campaign detail/dashboard and project map
- Historical context remains in: this workpackage and dated `work/evidence/` records
- Runtime/status evidence: work/evidence/sd-qualifier-aug10-campaign-inconclusive-closeout-20260813.md
- Cleanup deferred to: unrelated historical campaigns and pre-existing workspace changes
