# Optimize HPT USB CSV publication throughput Workpackage

Status: complete
Type: workpackage
Updated: 2026-07-26
Next Action: none; operator review proceeds on the live Graph screens

Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md
Parent: work/wp/active/wp-triple-mirrored-circular-hpt-log-store.md
Depends On: work/wp/completed/wp-optimize-hpt-triple-mirror-read-performance.md
Canonical Truth: docs/sd-card-operation-and-recovery.md

## Current Context

Final-flush timing on candidate `A7055AC3...74FA4` shows that HPT staging is
only `2.17` to `2.28 s` for approximately 1,700 records, while USB
publication takes `21.24 s` on A and `31.71 s` on B. The writer formats and
submits every CSV row through a separate `fx_file_write`, so a 90-day export
would issue roughly 129,600 small FileX writes. Linear end-to-end projections
are `28.84 min` on A and `42.81 min` on B.

The accepted safety contract writes `HPTE.TMP`, verifies the staged/published
record count and completion trailer, closes and flushes, renames only on full
success, and performs a final flush. All three HPT copies remain synchronously
validated before USB publication. Those behaviors are release constraints.

## Recommendation

Aggregate complete CSV rows in a bounded aligned RAM buffer and issue large
sequential FileX writes. Measure row formatting, buffered write calls/bytes,
close, flush, and rename separately. Preserve every current fail-closed
transition and the final publication sequence.

Use a conservative fixed buffer that fits existing internal-RAM headroom and
flush only between complete rows. A row larger than the buffer must be written
directly rather than truncated. Do not remove either media flush, publish the
final filename early, weaken the completion trailer/count check, change the
CSV contract, skip a mirror, format either card, or require favorable B media.

## Current State

Completed:

- Baseline B-first/A-second real exports and full timing are recorded in
  `work/evidence/hpt-usb-publication-timing-20260726.md`.
- The HPT writer coalesces complete rows in an aligned bounded 8 KiB buffer;
  the legacy writer remains unchanged.
- Phase telemetry reports construction, writes, close, flush, rename, write
  shape, and buffer capacity.
- All 21 check suites, all 8 Python model suites, `py_compile`, and the
  isolated clean build/package passed.
- The exact successor image passed B first and A second with real exports,
  preserved CIDs/topology, zero gaps/errors/fallbacks/drops, and both Graph
  screens restored for operator review.
- Results and updated projections are recorded in
  `work/evidence/hpt-usb-csv-buffered-publication-20260726.md`.

## Goal

The existing CSV and atomic-publication contract remain byte-compatible while
USB publication uses bounded large writes, reports phase costs, and materially
reduces the conservative 90-day duration on both installed USB sticks.

## Why It Matters

The current 29-to-43-minute projection is dominated by avoidable per-row
FileX overhead, not SD integrity work or raw USB bandwidth.

## Major Outcomes

- Far fewer FileX write calls without partial-row buffering.
- Exact timing attribution and B/A before/after evidence.
- No change to HPT format, CSV semantics, or atomic publication.
- Zero new gaps, drops, system errors, CID changes, or B-media normalization.

## Delivery Stages

Use stages when sequencing, parity gates, or handoff cost matter.

| Stage | Outcome | Entry Evidence | Exit Evidence |
| --- | --- | --- | --- |
| 1 | Writer baseline and contract | `A7055AC3...` A/B timing | exact bottleneck and invariants |
| 2 | Buffered implementation | source contract | focused models/static checks |
| 3 | Software candidate | implementation complete | all checks, clean build/package, hashes |
| 4 | Controller B proof | exact CID/split baseline | published export and phase timing |
| 5 | Controller A proof | B passes | published export and phase timing |
| 6 | Disposition | A/B measurements | projections, docs, Tool Shed closeout |

## Related Artifacts

- Evidence:
  `work/evidence/hpt-usb-publication-timing-20260726.md`
- Inventories:
  `work/inventories/smi-current-setup-a.md`,
  `work/inventories/smi-current-setup-b.md`

## Rough Sequence

1. Capture the exact writer and publication invariants.
2. Add a bounded aligned row buffer and phase telemetry.
3. Pass focused models, all checks, clean Debug build, package, and hash gate.
4. Program B first; preserve CID, `86/86/1`, and `2/1/0`; run a real export.
5. If B passes, program A and repeat.
6. Compare A/B gains, project 30/90 days, restore Graph, update docs, close.

## Milestones

### Milestone 1: Safe optimized candidate

Completion criteria:

- [x] CSV row content/order and completion trailer remain unchanged.
- [x] Temporary-file, count, close, flush, rename, final-flush, and failure
  contracts remain intact.
- [x] Buffer writes are bounded and never truncate or split a row unsafely.
- [x] Phase telemetry distinguishes formatting, write, close, flush, rename,
  write calls, bytes, and maximum write size.
- [x] Focused and repository-wide checks plus clean build/package pass.

### Milestone 2: Physical acceptance

- [x] B passes first with expected CID and preserved abnormal lanes.
- [x] A passes after B.
- [x] Both exports publish with zero gaps/errors/fallbacks/drops.
- [x] USB publication improves materially on both controls.
- [x] 30/90-day projections and any remaining bottleneck are explicit.

## Open Questions

- What buffer size gives the best gain without unacceptable USB-thread
  blocking latency? Start at 8 KiB and use measured phase data.
- Optional FileX preallocation is not justified: buffering reduced the
  conservative 90-day projection from `42.81 min` to `5.61 min`, while
  preallocation would add allocation and fault-tolerant FAT transaction risk.

## Completion Standard

This workpackage is complete when one exact buffered image passes all software
gates, publishes a real retained-range CSV on B then A without weakening the
accepted safety contract, preserves both CIDs and B's abnormal media, reports
phase timing, and records defensible long-range projections.

## Closeout Routing

- Current truth promoted to: `docs/sd-card-operation-and-recovery.md`
- Historical context remains in:
  `work/evidence/hpt-usb-publication-timing-20260726.md`
- Runtime/status evidence:
  `work/evidence/hpt-usb-csv-buffered-publication-20260726.md`
- Cleanup deferred to: any binary/compressed export or CSV-contract change
