# Controller A SD card read failure

Status: complete; immediate cause established and systemic hypotheses transferred to qualification
Type: incident
Updated: 2026-09-23
Next Action: none; retained as history; any new execution requires a new explicit owner instruction
Campaign Reason: Historical record from campaign 011-qualify-degraded-sd-operation-and-logging-integrity; retained independently after campaign closure and requires no separate owner-queue execution.
Campaign: standalone
Canonical Truth: docs/sd-card-operation-and-recovery.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Impact

Controller A produced a syntactically valid USB CSV whose requested end time was
2026-07-22 05:19:52, but whose last record was 2026-06-17 10:59:00. The file
contained 78,100 continuous one-minute records and silently omitted the final
50,060 requested minutes.

After power-off removal, the SD card repeatedly failed reads on Linux and
Windows. It must not be returned to service.

## Evidence

- Linux through the Norelsys reader, both through the USB hub and directly on
  `srv-rpi00`, reported device resets and read errors near sectors 14,336-14,338.
- An independent Windows reader reported hardware I/O errors at logical blocks
  `0x3840`-`0x3850` (14,400-14,416).
- GNU ddrescue recovered almost all card bytes but confirmed isolated unreadable
  512-byte sectors near byte offsets `0x00700000`, `0x00704000`, `0x00708000`,
  `0x0070FE00`, `0x00BE0000`, `0x04AE0000`, `0x04AEFE00`, and `0x04AFFE00`.
- The recovered BPB definitively identifies FAT32: 512-byte sectors, eight
  sectors/4 KiB per cluster, two reserved sectors, one FAT, 15,160 FAT sectors,
  15,523,840 total sectors, and 1,938,584 data clusters.
- The first FAT occupies sectors 2-15,161. The repeatable failures around sector
  14,336 are therefore inside the card's only FAT copy.
- FileX fault-tolerant metadata was present, but it cannot repair an unreadable
  NAND/flash sector.
- A/B log mirrors share the same FAT and physical media; they do not protect
  against this failure class.

Original recovery artifacts on `srv-rpi00`:

- `/var/tmp/smi-control-a-failed-sd-20260722.img`
- `/var/tmp/smi-control-a-failed-sd-20260722.map`
- `/var/tmp/smi-control-a-failed-sd-20260722-recovered-files/`
- `/var/tmp/smi-control-a-failed-sd-20260722-recovered-files.sha256`
- `/var/tmp/smi-control-a-failed-sd-20260722-carved-records.bin`
- `/var/tmp/smi-control-a-failed-sd-20260722-carved-records.csv`

Compact durable preservation is complete. The retained archive, matching
source/destination SHA-256 hashes, access route, and explicit decision not to
duplicate the 7.95 GB raw image are recorded in
`work/evidence/controller-a-failed-sd-compact-preservation-20260725.md`.
Per the owner's direction, the original raw `.img` was deleted from
`srv-rpi00.local` after compact-archive verification; the other compact source
artifacts were retained.

## Why the USB export was silently truncated

`loadExportLogDataToSdram()` loops only while `gSdCardState` remains
`SD_CARD_STATE_READY`. A media error can terminate the traversal before the
requested end time. The function nevertheless sets
`gIsNewSaveLogRecordsFromSdToUsb = true`, so the USB thread writes whatever
prefix reached SDRAM using a header that still declares the requested end.

The observed continuous prefix followed by a missing tail is consistent with
that control flow and the independently proven card read failure.

## Why services 470 and 474 did not predict it

- Service 474 observes normal logger outcomes but does not read every physical
  sector or every FAT sector.
- The qualified service-470 run used a 64 MiB budget. Repeated success proves
  only the allocated test files and metadata paths it exercised.
- FAT allocation tends to reuse the earliest available clusters. A bounded test
  area does not force access across the complete 7.4 GiB address space or the
  entire FAT.
- A controller can continue opening known files while a currently unused FAT
  sector elsewhere is unreadable. A host mount/free-space scan is more likely
  to read the complete FAT and expose it.

## Likely mitigations

### Priority 1: fail closed on incomplete export

- Track whether date traversal reached the requested end normally.
- Treat SD state changes, `LoadLogRecords` I/O status, or premature loop exit as
  export failure.
- Do not queue USB serialization for an incomplete load.
- Remove or clearly mark any partial file and report the actual failure/status
  to the operator.
- Add a regression test that injects an SD read failure after a valid prefix and
  proves no apparently successful CSV is produced.

### Priority 2: add non-destructive full-address read coverage

- Add an engineering/background scrub that reads every physical sector in
  bounded low-priority slices while normal logging continues.
- Persist progress and first failing LBA so the sweep resumes after restart.
- Rate-limit the sweep and yield to logger, chart, export, and control work.
- A complete pass, not runtime alone, is the acceptance unit.

### Priority 3: broaden allocation/write coverage

- Increase the endurance test budget beyond the current 64 MiB when free space
  permits, while preserving an explicit safety reserve.
- Allocate, verify, delete, and reallocate test extents so FAT entries and free
  space beyond the first small region are exercised.
- Report address/cluster coverage, not only record counts.
- Full-media write coverage remains destructive unless constrained to free
  filesystem space; qualify expendable cards separately with destructive
  full-capacity write/read cycles.

### Priority 4: reconsider one-FAT formatting

- The formatter currently requests one FAT. Two FAT copies would provide a
  second metadata copy for recovery tooling and some failure scenarios.
- Confirm FileX behavior when FAT 1 has a hard read error before relying on FAT
  2 automatically; two FATs reduce risk but do not fix failing media.
- Preserve FileX fault-tolerant journal compatibility and repeat power-cut tests
  if geometry changes.

### Priority 5: media policy

- Retire this card and preserve it as the failure specimen.
- Qualify multiple samples of the intended industrial card model using complete
  read sweeps plus destructive full-capacity write/read cycling on expendable
  media.
- Define replacement/lifetime policy; SD cards expose no dependable SMART-style
  health metric to this firmware.

## Current conclusion

The direct cause of the July 22 data loss was physical unreadability in several
card regions, including the only FAT. The firmware did not create that evidence
by formatting as FAT16; the recovered BPB proves FAT32. The correctable firmware
contribution was fail-open export behavior: an SD traversal failure became a
plausible-looking partial success. That behavior and the whole-address coverage
gap have been corrected in the final candidate.

The available evidence cannot identify one initiating mechanism. It does not
prove that controller power/interface behavior, accelerated metadata activity,
media population, formatting, or an independent card defect caused the physical
failure. The original raw recovery image is no longer retained, and one failed
specimen cannot discriminate those hypotheses.

The causal investigation is therefore closed with `initiating mechanism
indeterminate`. Systemic risk is not waived: the active HPT media qualification
work owns CID-linked baseline/post full-address scrubs, multiple samples from at
least two retail families, Controller A/B comparison, scheduled interruption
coverage, and escalation to power-rail/signal-integrity measurement if failures
cluster by controller. Workload analysis escalates only if failures cluster
under accelerated activity across controllers.

## Owner-directed reconciliation — 2026-09-23

Reconcile stale active lifecycle to the previously documented completed disposition; no new test or broader acceptance claim.

Authorization: Owner instruction in this conversation on 2026-09-23 to close/abandon the named work and mitigate older active records.

Previous header retained for evidence interpretation:

> Status: complete; immediate cause established and systemic hypotheses transferred to qualification
> Next Action: none; systemic-risk discrimination is owned by the active multi-controller, multi-sample HPT qualification workpackage

Earlier observations, unchecked historical criteria, and procedures below/above remain history. This administrative disposition does not qualify a new firmware candidate, physical card, thermal system, ECO, or release. Current 00.05.01 obligations in IDEA-0006, IDEA-0007, MAP-0009, and CAMP-0028 remain unchanged.
