# Spike: Raw HPT boot reports no SD card

Status: complete; startup ownership and diagnostic-hold false-warning paths corrected and visually verified
Type: spike
Updated: 2026-09-23
Next Action: none; retained as history; any new execution requires a new explicit owner instruction
Campaign Reason: Historical record from campaign 011-qualify-degraded-sd-operation-and-logging-integrity; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Parent: work/wp/active/wp-triple-mirrored-circular-hpt-log-store.md

## Question

Why does Controller A report that no SD card is installed after booting with a
known HPT 2.0 qualification card, and does that message indicate an unreadable
or corrupt card or an obsolete FAT32/FileX state mapping?

## Timebox

One engineering session for read-only live-state capture, source tracing, a
bounded ownership/state fix, software gates, Controller A reflash, and physical
power-cycle verification. Do not initialize or format the card during
diagnosis.

## Findings

- The visible startup message is real, but it does not describe the media.
  Read-only J-Link capture through `srv-rpi00` on 2026-07-23 found:
  - `gSdCardState = SD_CARD_STATE_UNAVAILABLE` (`3`);
  - `gIsSdCardAvailable = false`;
  - `gHptMediaStatus.State = HPT_MEDIA_STATE_HPT_READY` (`5`);
  - HPT status and SSP status both zero;
  - sector count `0x00ECE000`;
  - qualification state `HPT_LOG_QUALIFICATION_STORE_OPEN` (`3`);
  - `store.is_open = true`, `store.qualification_media = true`;
  - all three lanes present (`last_lane_mask = 7`);
  - media generation 4, oldest sequence 145, and next sequence 157; and
  - no HPT storage-error or replace-media flag.
- The capture proves the card and open HPT store were valid after the observed
  restart/reset. It does not prove physical power-loss recovery: J-Link
  `69664082` remained electrically connected and back-powered Controller A
  while external power was removed. The attempted cut is invalid as physical
  power-cut evidence.
- The public SD state contradicts the HPT owner state. The FileX open failure
  expected on raw HPT media invokes `g_fx_media_sd_err_callback()`, which
  unconditionally publishes `SD_CARD_STATE_UNAVAILABLE` and sets the SD system
  error bit. A late callback can overwrite the `SD_CARD_STATE_RAW_HPT` value
  published after the successful HPT probe.
- Once overwritten, the SD loop enters the non-ready retry branch, repeatedly
  attempts FileX open on media already owned by the live raw-HPT session, drops
  queued log records, and never repeats HPT classification. This is both a
  false message and an ownership/liveness defect.
- `serviceSdMissingHomeNotice()` only treats legacy FileX
  `SD_CARD_STATE_READY` as present. It does not re-arm/reset for
  `SD_CARD_STATE_RAW_HPT`, and it can count the expected transient FileX
  failure while HPT classification is still in progress.
- The earlier version-5 media spike already requires missing media to be
  distinguished from present but non-FAT32 media. The live evidence confirms
  that requirement is not yet met in this startup path.
- The bounded correction was committed as `548aabc3` and programmed/verified
  on Controller A from Debug S-record SHA-256
  `4BEDE424D276796271DDB275787ED1D02ED2205B3A8616FF631C01141602EB60`.
- Post-program read-only capture found:
  - `gSdCardState = SD_CARD_STATE_RAW_HPT` (`6`);
  - `gSdCardLastStatus = FX_SUCCESS`;
  - the SD system-error bit clear;
  - HPT ready with the store open and all three lanes present; and
  - next sequence 160, advanced from 157 before programming, proving normal
    queued minute logging resumed after boot.
- The corrected candidate passed 74 HPT commissioning invariants, 17 customer
  export invariants, 20 SD safety invariants, and isolated clean Debug and
  Release builds.
- In a later armed attempt the operator disconnected power from J-Link, then
  removed controller power, made the display go dark, restored controller
  power, waited for boot, and only then powered J-Link again. Read-only
  post-reconnection evidence showed that normal MCU startup clearing did not
  occur:
  the arm magic, `HOLDING_CUT_POWER` state, and diagnostic device wrapper all
  survived in volatile RAM while the controller run delta advanced from 4 to
  562. Those objects are ordinary `.bss` inside the startup-cleared range, and
  powering J-Link later cannot recreate them. The evidence does not show that
  the operator performed the sequence incorrectly; a hold-up/brownout path or
  qualification-method flaw remains unresolved. The visible
  `SD card is not installed` warning in that attempt was the
  harness's intentional `UNAVAILABLE` hold state, not a startup classification
  result and not evidence against commit `548aabc3`.
- Cancelling that retained hold restored `RAW_HPT`, normal device ownership,
  HPT/SSP status zero, and logging with all 15 queued records completed and
  zero drops. Cancellation also allowed interrupted sequence 188 to commit, so
  that attempt cannot be used for recovery qualification.
- The target helper now requires a successful `VerifyPostCutBoot` capture
  before recovery. It rejects retained arm command/state, retained diagnostic
  device ownership, an unhealthy raw reopen, changed media generation, or a
  tail that does not stop at the interrupted sequence.
- A RAM-only reset witness subsequently passed: after J-Link power-off, a
  10-second discharge wait, controller power-off, 30 seconds with both
  unpowered, controller boot with J-Link off, and later J-Link power-up, both
  volatile markers were zero, SD state was `RAW_HPT`, and HPT media/status were
  ready/zero. This proves the operator sequence executes normal startup and
  that later J-Link power-up does not recreate diagnostic RAM.
- The next armed cut nevertheless retained the arm command, hold state, and
  diagnostic wrapper through the same reported timed sequence while the
  powered-off target cable remained attached. The automatic gate rejected it,
  and cancellation restored healthy `RAW_HPT` logging. For armed qualification,
  probe power-off alone is no longer accepted; the target/SWD cable must be
  physically removed before controller power-off.
- Source tracing then confirmed that `serviceSdMissingHomeNotice()` is a
  periodic Home-screen check, not a watchdog symptom. Its three-count delay is
  reached in about three seconds while Home is active. The diagnostic harness
  itself had published `SD_CARD_STATE_UNAVAILABLE` while holding the SD thread
  at the armed boundary, so the periodic routine correctly displayed the
  warning for the state it was given even though the raw store was healthy.
- Commit `9849b00e` removed that unnecessary public-state mutation. The private
  diagnostic state remains `HOLDING_CUT_POWER`, while public media state remains
  `SD_CARD_STATE_RAW_HPT`. The change passed 81 HPT commissioning invariants,
  17 customer-export invariants, SD safety checks, and clean Debug/Release
  builds. Controller A was programmed and verified with Debug S-record SHA-256
  `46AFA0139C06C83DC8FEEFC556F7C80FF044F3A913E30837F71040D89A77D79B`.
- Corrected attempt 5 held at data-durable-before-commit with public
  `RAW_HPT`, HPT ready/status zero, SD last status zero, and no system error.
  After the hold had remained active substantially longer than the periodic
  warning delay, the operator visually reported `no warning`. This closes the
  false-warning question independently of the still-pending physical
  interruption/recovery qualification.

## Recommendation

Keep FileX and raw-HPT ownership mutually exclusive and publish availability
from the active owner:

1. Mark the state as opening/probing during the bounded HPT classification.
2. When HPT opens successfully, publish `RAW_HPT`, clear the expected stale
   FileX status/system-error indication, and leave `gIsSdCardAvailable` false
   because that variable means FileX owns the SD media.
3. Make the FileX error callback ignore a late FileX notification when a
   validated raw-HPT store is already open; it must not revoke raw ownership.
4. Make the Home notice treat both FileX-ready and raw-HPT-ready as installed,
   and suppress the missing-card countdown while HPT classification is active.
5. Add source gates for those state/ownership invariants.
6. Establish a true controller-off method and repeat cold boot and interrupted
   write recovery using the same card. Acceptance requires the volatile-RAM
   reboot gate to pass, raw-HPT state to remain stable, no false missing-card
   dialog, queued logging to continue, and the recovery harness to pass.

## Follow-Up

- [x] Capture live state without formatting, initialization, or recovery append.
- [x] Confirm HPT media is valid and the three-lane store reopened.
- [x] Trace the contradictory state to the legacy FileX callback/retry path.
- [x] Implement the ownership and Home-notice corrections.
- [x] Add and pass source-level regression checks.
- [x] Build Debug and Release with the normal clean isolated workflow.
- [x] Program Controller A and verify raw-HPT boot state read-only.
- [x] Run a harmless volatile-witness power-cycle using the same operator
  sequence before arming another media write.
- [x] Prove volatile RAM is reset when the explicit 10-second J-Link discharge
  and 30-second both-off waits are used.
- [x] Confirm visually that no missing-card warning appears during the corrected
  armed hold.
- [x] Hand the remaining data-durable physical cut and recovery qualification
  to
  `work/checklists/checklist-controller-a-hpt-wrap-and-power-interruption-qualification.md`.

## Owner-directed reconciliation — 2026-09-23

Reconcile stale active lifecycle to the previously documented completed disposition; no new test or broader acceptance claim.

Authorization: Owner instruction in this conversation on 2026-09-23 to close/abandon the named work and mitigate older active records.

Previous header retained for evidence interpretation:

> Status: complete; startup ownership and diagnostic-hold false-warning paths corrected and visually verified
> Next Action: none; physical power-cut recovery continues under the linked Controller A checklist

Earlier observations, unchecked historical criteria, and procedures below/above remain history. This administrative disposition does not qualify a new firmware candidate, physical card, thermal system, ECO, or release. Current 00.05.01 obligations in IDEA-0006, IDEA-0007, MAP-0009, and CAMP-0028 remain unchanged.
