# HPT bounded-export pre-implementation baseline and budget proposal

Status: accepted
Result: approved; implementation authorized
Type: evidence
Updated: 2026-08-14
Next Action: implement and verify the approved range-addressable correction
Campaign: 003-complete-bounded-hpt-slow-export-firmware-ticket
Parent: work/tickets/ticket-eliminate-full-history-scan-latency-for-bounded-hpt-customer-exports.md
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md

## Scope and dependency gate

This is the pre-implementation evidence required by campaign
`003-complete-bounded-hpt-slow-export-firmware-ticket`. It reviews the current
source, deterministic models, canonical requirements, and the preserved
Controller A/B measurements. It does not program a controller, issue a
controller command, modify an SD card, operate a USB drive, deploy software, or
change the terminal SD Qualifier campaigns.

The dependency checkpoint exists as commit
`80419cb78b598e5634a1ded670745ebe387a83f1`. Its exact 31-file manifest was
verified before campaign 002 completed, and no product source was included.

## Current path and phase baseline

`beginHptCustomerExport()` snapshots the media generation, oldest retained
sequence, and exclusive end sequence, then starts the shared triple-circular
iterator at `oldest_sequence`. `serviceHptCustomerExport()` reads and votes each
available record through the exclusive snapshot end, validates its payload and
gap context, and stages only customer records whose timestamps are inside the
requested range. USB publication begins only after a clean iterator end.

The preserved 2026-08-11 target telemetry separates the phases as follows:

| Phase | Controller A | Controller B | Current scaling |
| --- | ---: | ---: | --- |
| Range location | none; iterator starts at sequence 1 | none; iterator starts at sequence 1 | no addressable lookup exists |
| HPT read plus mirror validation | 1,064,047 records; 206,052 commands; 6,390,522 blocks | 623,996 records; 136,122 commands; 3,762,984 blocks | linear in all retained history |
| Selection and row staging | 9 rows within the traversal | 10 rows within the traversal | linear in selected rows after the scan |
| Selection elapsed | 1,177.33 s | 693.90 s | about 904 and 899 scanned records/s |
| USB publication | 0.87 s; one 1,133-byte write | 0.16 s; one 1,183-byte write | separate final phase |
| Total | 1,178.22 s | 694.07 s | selection dominates |

The present telemetry couples HPT reads, quorum resolution, record-envelope
validation, gap accounting, and row selection inside one traversal timer. The
read command/block counters and selected-row count bound those components even
though they do not yet have separate elapsed timers.

The current complexity is `O(N retained records)` HPT reads and validation plus
`O(R selected records)` staging/publication. The requested ten-minute ranges
therefore took 11.6 to 19.6 minutes even though USB required less than one
second.

## Invariants that the correction must preserve

| Invariant | Required range-addressable behavior |
| --- | --- |
| Snapshot | Pin media generation and exclusive sequence end; never include a concurrent append beyond that end. |
| Retention/reclamation | Reject a range that is no longer fully retained and fail if reclamation overtakes any selected or proof range. |
| Sequence gaps | Preserve snapshot-gap and requested-range-gap accounting, including gaps whose neighboring timestamps straddle the request. |
| Three-lane voting | Use the existing majority/degraded rules for every returned record; never trust an unverified index as customer data. |
| Malformed or ambiguous copies | Fail closed on payload-envelope mismatch or multiple disagreeing degraded copies. |
| Wrap and retirement | Resolve logical sequence spans across independently advanced and retired lane segments without relying on physical order. |
| Timestamp order | Include duplicates and forward/backward-adjusted timestamps exactly; no monotonic-timestamp binary search is permitted. |
| Capacity | Reject before USB temporary-file creation as soon as the selected count exceeds `MAX_EXPORT_LOG_RECORDS`. |
| Media identity | Abort on media-generation change; an index belongs to exactly one media generation and sequence snapshot. |
| Telemetry/UI | Report lookup, HPT reads, validation, staging, USB phase, progress, terminal reason, and the request ID truthfully. |
| USB compatibility | Leave the accepted 32 KiB FileX cap and standards-based USB publication lifecycle unchanged. |

## Design alternatives

### A. Persisted segment time envelopes plus a bounded range directory — recommended

Add an explicitly versioned HPT media feature that records checksummed first,
last, minimum, and maximum customer timestamps for each sealed logical segment.
Build a small sequence-ordered directory from those envelopes during the
existing open/header scan. Keep the active-segment envelope in RAM and recover
it by a bounded active-prefix scan after reboot.

Range lookup uses the conservative timestamp envelope to identify candidate
logical sequence spans. It includes boundary neighbors needed for gap proof,
merges independently advanced lane spans conservatively, and then runs the
existing three-lane read/vote/payload checks over every candidate record. The
index can exclude a segment only when its checksummed metadata proves the
segment cannot contain or bridge the requested range. Missing, conflicting, or
unsupported envelope metadata fails closed; it never becomes permission to
skip data.

This is the smallest reviewed direction that is addressable without assuming
timestamp monotonicity. Because existing media 2.0 does not contain the
envelopes, the feature must be explicit and must use the established
reinitialization/successor-qualification path; it must not silently rewrite or
pretend to upgrade existing cards.

### B. Rebuild a RAM-only full-history index after every boot — not recommended

This avoids a media feature but must reread and validate the retained history
after every reboot. On the measured million-record Controller A history the
rebuild would retain roughly the same 19.6-minute I/O burden, delay readiness,
and lose the index on the next restart. Background execution moves rather than
removes the cost and leaves early exports without the requested bound.

### C. Scan only a fixed suffix or increase bulk size — rejected

The existing bulk-16 experiment already reduced commands about 31x; bulk-8 and
bulk-16 had nearly identical elapsed time, showing that transfer and validation
now dominate. At the measured A/B rates, scanning only the 142,560-record
90-day-plus-event window still projects to approximately 158 seconds. A
sequence suffix alone also cannot prove exact inclusion for forward/backward
timestamp adjustments unless customer visibility is redefined, which this
ticket does not authorize.

## Proposed owner-approved latency budget

For a ten-minute customer range against a snapshot containing at least
1,000,000 retained records, on both Controller A and Controller B:

1. Terminal `PUBLISHED / COMPLETE` must occur within **60.0 seconds** from the
   accepted request when the already-mounted representative USB device remains
   healthy.
2. Range lookup plus HPT validation and SDRAM staging must complete within
   **30.0 seconds**; USB queue wait/publication receives the remaining
   **30.0 seconds** and remains separately reported.
3. The ordinary, duplicate, boundary, one-forward-jump, and one-backward-
   adjustment ten-minute fixtures may scan no more than **8,192 HPT records**.
   A malformed, missing, ambiguous, reclaimed, or generation-changed candidate
   must fail within the same lookup/read bound.
4. Near-capacity exports are not required to meet the ten-minute-range
   60-second budget. Their lookup cost must remain bounded independently of
   unrelated history, while validation, staging, and publication may scale
   with the records actually selected.
5. The budget is a product acceptance ceiling, not permission for
   controller-, SD-card-, or USB-specimen-specific behavior.

This keeps the ticket's initial 60-second customer target while making the
internal lookup/selection obligation testable and preserving a separate USB
compatibility phase.

## Pre-implementation checks

The unchanged candidate passed on 2026-08-14:

- `python script/test-hpt-customer-export-model.py`;
- `script/check-hpt-customer-export.ps1` — 34/34 invariants;
- `script/check-graph-data-policy.ps1` — all graph/export policy checks.

These checks confirm the baseline fail-closed behavior. They are not evidence
that the latency defect is corrected.

## Exact owner decision required

Approve the five-part budget above, or provide replacement limits for the
60.0-second total, 30.0-second lookup/selection, and 8,192-record ordinary
fixture scan ceilings. After approval, implementation and deterministic oracle
coverage may begin. Physical Controller A/B work remains a later, separate
exact-image/target/procedure authorization gate.

## Owner decision

Approved by the owner in the active campaign thread on 2026-08-14. The
approved limits are exactly the five-part budget above: 60.0 seconds total,
30.0 seconds for lookup/validation/staging, and at most 8,192 HPT records
scanned for the ordinary ten-minute fixtures against at least 1,000,000
retained records on both controller variants. Implementation is authorized;
controller programming and physical target execution are not yet authorized.
