# Incident: USB09 false write failure and USB compatibility architecture correction

Status: complete
Result: resolved
Type: incident
Updated: 2026-08-01
Next Action: none; retain as the architectural incident that produced the completed standards-led broad baseline
Campaign: 003-complete-bounded-hpt-slow-export-firmware-ticket
Parent: work/maps/map-smi-firmware-stabilization-and-qualification.md
Canonical Truth: docs/firmware-update-validation-and-recovery.md and docs/sd-card-operation-and-recovery.md
Resolved By: work/wp/completed/wp-restore-broad-usb-compatibility-baseline.md

## Impact

Controller A reported `USB write error; check or replace the USB drive` for
canonical drive `USB09` even though the final exported CSV was complete,
readable, structurally consistent, and durable after physical transfer to the
Raspberry Pi. The false terminal result undermines operator trust and shows
that the USB compatibility work completed in commit
`e31e01988ceb8498f2d474e131451a50665f6623` was overfit to prior specimens and
introduced a nonstandard mandatory durability sequence into the production
export path.

No firmware change was made during this incident. Controller A remained on the
qualified Release S-record SHA-256
`678BCC0BAEE940D27417DC889C296732281B7AD40F04EC952DB620D7BC6F40FD`.

## Timeline

- 2026-07-30: Controller A export request 2 failed with raw status `0x5C` at
  `HPT_EXPORT_USB_PHASE_FILE_CLOSE` after 131,027 requested bytes. The JSON
  decoder displayed that value as decimal `92`; it was later misdocumented as
  hexadecimal `0x92`/`FX_ERROR_FIXED`. The close call consumed exactly 50
  seconds. Read-only Pi
  inspection found an inconsistent `/HPTE.TMP`, differing FAT copies, and an
  incorrect free-cluster summary.
- USB09 was reformatted as FAT32 on Windows. The new filesystem UUID was
  `24E4-9846`; the drive retained label `USB09`, 7.5 GiB capacity, USB serial
  `2512300955508200677718`, and USB/SCSI identity `General UDisk`, ANSI level
  2.
- The Pi then passed a controller-like 131,027-byte create, write, flush,
  rename, unmount/remount, size, and SHA-256 verification in 268 ms. The test
  file was removed. Linux reported only the benign Windows convention of
  storing the volume label in the root directory rather than duplicating it in
  the boot sector.
- 2026-07-30: Controller A export request 3 selected 7,452 healthy HPT records
  with no gaps and wrote 374,707 bytes in 12 calls. Write, close, flush, rename,
  and initial verification completed. The firmware then deliberately issued
  STOP UNIT, cut USB VBUS, re-enumerated the drive, and failed its fresh-session
  verification with raw status `0x5C`.
- Read-only Pi inspection after request 3 found a structurally clean FAT32
  filesystem and a complete exported file:
  `SMI6_07162026_20260725t0836-20260730t1541.csv`.
- The CSV size was exactly 374,707 bytes. Its SHA-256 was
  `d675af49dde995139ee12e734570e76d7d33ad5f95c3fd310876c0243821b951`.
  Its final marker was
  `Export completion: COMPLETE; records published: 7452`.

## Cause

### Immediate failure

The production export path treated a raw status value of `0x5C` returned somewhere
inside the custom post-power-cycle verification sequence as an authoritative
USB publication failure. The existing telemetry collapses several post-remount
operations into `HPT_EXPORT_USB_PHASE_FRESH_VERIFY`, so the exact returning
call is not yet proven. The prior JSON printed decimal `92`, which was
incorrectly transcribed as hexadecimal `0x92`. The binary evidence contains
`5C 00 00 00`; this is not FileX `FX_ERROR_FIXED` (`0x92`). The status
namespace and exact post-remount call site remain unproven.

The exported content itself did not fail. Exact size, completion marker,
full-file read, SHA-256, FAT traversal, removal, and reinsertion all passed on
the Pi.

### Architectural cause

The compatibility release added roughly 2,441 lines to
`src/Thread_Usb_entry.c` plus extensive modifications inside vendored SSP
USBX storage sources. It made this custom sequence mandatory after a normal
FileX publication:

1. compute a full-file CRC;
2. issue SCSI `START STOP UNIT` with STOP MEDIA;
3. wait a compatibility-dependent quiescence interval;
4. disable USB VBUS;
5. require teardown of USBX/FileX objects;
6. hold VBUS off for five seconds;
7. restore power and require a new media session;
8. flush recovery state;
9. verify size and completion marker;
10. reread the entire file and compare CRC.

That sequence is a useful destructive/interruption qualification test, but it
is not the documented baseline production write workflow. Making it part of
every customer export increases the command surface, depends on optional SCSI
behavior, and turns a valid durable export into an operator-visible failure.

The earlier development method was reactive: test available specimens, add
fallbacks for observed failures, then declare compatibility. That cannot cover
future drives and must not remain the design method.

## Recovery

At the initial recovery checkpoint no firmware recovery had been deployed. The
valid CSV was confirmed on the Pi, USB09 was safely unmounted after read-only
inspection, and the incident remained active because that production firmware
still reported the false failure.

Evidence summaries:

- `work/evidence/hpt-log-store-circular-mirror/controller-a/usb-incidents/new-drive-write-error-20260730T153228/`
- `work/evidence/hpt-log-store-circular-mirror/controller-a/usb-incidents/usb09-retry-after-reformat-20260730T154300/`

On 2026-07-31, USB09 passed a fresh read-only Pi pretest before regression.
Its label, UUID, serial, capacity, clean FAT state, and historical export hash
matched the incident record. A complete 8,053,063,680-byte direct read and
kernel-error gate passed, and the drive was left unmounted. Evidence is in
`work/evidence/usb09-pi-pretest-health-20260731.md`.

The recovery candidate then reproduced the request-2 failure before any
optional power/remount behavior. USB09 mounted on first insertion with a valid
session. A full retained-range export selected 8,540 healthy records with no
gaps, requested 99,328 bytes in 97 writes capped at 1,024 bytes, and then
`fx_file_close()` consumed exactly 50 seconds and returned raw `0x5C`.
Publication failed while the drive remained mounted and the session valid.
Evidence is in
`work/evidence/usb09-recovery-candidate-controller-failure-20260731.md`.

Independent no-repair Pi inspection confirmed that this recovery-candidate
failure was not a false terminal result. The two intact FAT copies differ, the
free-cluster summary is wrong, `USB09_RCV.csv` is absent, and `HPTE.TMP` is
97,280 bytes rather than the 99,328 requested bytes. It ends mid-record without
a completion marker. The prior valid export is unchanged and the Pi logged no
transport errors. Evidence is in
`work/evidence/usb09-recovery-candidate-pi-failure-inspection-20260731.md`.

Raw `0x5C` is USBX `UX_TRANSFER_TIMEOUT`; the stock USBX FileX driver returns
that transport value without translating it into the FileX namespace. The
exact 50-second close duration matches `UX_HOST_CLASS_STORAGE_TRANSFER_TIMEOUT`.
The 1 KiB cap is disproven as the compatibility baseline. Because the same
clean USB09 specimen previously completed a larger 32 KiB export in 12 calls
and closed in 0.01 seconds, the next correction is an isolated 32 KiB HPT
FileX cap with no transport fork, timeout replay, VBUS action, or drive rule.
That isolated candidate now passes the focused software gates, canonical
managed Debug and Release rebuilds, packaging, and exact artifact freezing at
source commit `8fa1479b6b942f72f18ff4ed2cb1a5c73ddc2501`. Evidence is in
`work/evidence/usb-32k-recovery-software-candidate-20260731.md`.
The exact Release candidate was subsequently deployed B first and A second;
both full-flash readbacks and complete post-reboot SD identity/health gates
pass. Evidence is in
`work/evidence/usb-32k-recovery-controller-deployment-20260731.md`.
After explicit authorization and a full Windows FAT32 format, USB09 passed
exact Pi identity, equal-FAT, filesystem, full-device direct read, capacity,
content, unmounted-state, and kernel-error gates. Evidence is in
`work/evidence/usb09-post-windows-format-pi-health-20260731.md`.
USB09 then passed the identical Controller A failure-range regression on the
deployed isolated 32 KiB candidate: 8,540 healthy records, no gaps, 429,175
published bytes in 14 writes, zero close delay, 2.12-second USB time, and zero
errors. Evidence is in
`work/evidence/usb09-32k-recovery-controller-pass-20260731.md`.
Independent Pi verification passed equal FATs, filesystem state, exact
429,175-byte size, all 8,540 sequential records, range/health metadata,
completion marker, absence of temporary artifacts, and kernel-error gates.
Evidence is in `work/evidence/usb09-32k-recovery-pi-pass-20260731.md`.

## Final Resolution

USB09 passed the isolated 32 KiB Controller A regression and independent Pi
validation. All eight four-drive A/B observations then passed. The completed
architecture correction removed mandatory optional-command, VBUS-cycle,
remount, and full-reread behavior from production; proved the bounded
once-per-attachment enumeration recovery; added strict BOT CSW validation; and
passed representative cross-device delta regression without specimen-specific
behavior. The settled production policy is promoted in
`docs/firmware-update-validation-and-recovery.md`.

The current closeout is recorded in
`work/wp/completed/wp-restore-broad-usb-compatibility-baseline.md` and
`work/evidence/usb-bot-representative-delta-regression-20260801.md`.

## Prevention

Use published specifications and vendor-supported architecture as the design
baseline:

- Renesas USBX Host Class Mass Storage Module Guide:
  https://www.renesas.com/en/document/apn/usbx-host-class-mass-storage-module-guide
- USB-IF Mass Storage Class Bulk-Only Transport 1.0:
  https://www.usb.org/sites/default/files/usbmassbulk_10.pdf
- Eclipse ThreadX USBX upstream:
  https://github.com/eclipse-threadx/usbx
- Eclipse ThreadX FileX upstream:
  https://github.com/eclipse-threadx/filex
- INCITS T10 SCSI command references:
  https://www.t10.org/lists/2op.htm

Target production architecture:

1. Use the ordinary FileX publication lifecycle:
   temporary create/open, bounded sequential writes, close, media flush,
   destination replacement/rename, and final media flush.
2. Remove mandatory STOP UNIT, USB VBUS cycling, re-enumeration, and
   fresh-session CRC reread from normal exports. Retain them only in explicit
   qualification and fault-injection profiles.
3. Inventory every command the controller can issue, identify the governing
   USB-IF/T10 requirement and whether it is mandatory or optional for each
   supported subclass, and define request, response, timeout, retry, sense, and
   recovery behavior before implementation.
4. Implement Bulk-Only Transport recovery as specified: validate CBW/CSW,
   tags, residue, and status; clear the correct stalled endpoint; and perform
   Mass Storage Reset followed by clearing both bulk endpoints for invalid CSW
   or phase error.
5. On command failure, capture and classify `REQUEST SENSE` key/ASC/ASCQ.
   Cache unsupported optional commands per mounted media session. Do not
   repeatedly issue optional commands known to return `ILLEGAL REQUEST`.
6. Keep USB transport, USBX, SCSI, and FileX status namespaces separate in
   telemetry. Record the exact operation that produced each status.
7. Preserve content integrity with exact length, completion marker, close,
   flush, atomic publication where supported, and optional CRC/manifest.
   Continue to fail closed on unreadable content, size/marker/CRC mismatch,
   transport failure without successful recovery, or actual filesystem error.
8. Compare the SSP 2.6.1 USBX/FileX implementation against current upstream
   releases and selectively backport reviewed host-storage fixes without
   blindly replacing the Renesas HCD integration.
9. Build deterministic host-side tests for command failure, malformed CSW,
   residue mismatch, stalls, phase errors, timeouts, unit attention, not-ready,
   medium error, write protection, unsupported SYNCHRONIZE CACHE, and
   unsupported START STOP UNIT.
10. Use USB packet capture or USBX transport tracing for unexplained failures
    instead of inferring the command that failed from a high-level timeout.

## Follow-Up Disposition

- [x] Completed the standards-led correction under a successor workpackage.
- [x] Produced the controller command/transaction inventory in
  `work/inventories/inventory-usb-host-transactions-and-compatibility-policy.md`.
- [x] Traced raw status `0x5C` to the FileX/USBX transport timeout boundary.
- [x] Separated normal production publication from optional qualification.
- [x] Removed optional cache/STOP behavior from production, so no production
  per-device capability cache is required; valid class/sense failures remain
  fail-closed.
- [x] Compared the vendored storage stack with applicable upstream behavior
  without authorizing an SSP migration.
- [x] Added deterministic transport tests and USB09 regression coverage.
- [x] Passed canonical builds, B-first/A-second deployment, the complete matrix,
  enumeration recovery, BOT validation, and representative delta regression.
- [x] Promoted the settled production architecture into current documentation.
