# Ticket: Eliminate full-history scan latency for bounded HPT customer exports

Status: complete
Result: complete; A/B million-history performance, successor, and independent-reader checks pass; physical wrap test deferred by owner
Type: ticket
Updated: 2026-08-17
Next Action: none; retain the million-record datasets and carry destructive physical wrap testing only in later qualification
Campaign: 003-complete-bounded-hpt-slow-export-firmware-ticket
Parent: work/maps/map-smi-firmware-stabilization-and-qualification.md
Canonical Truth: docs/sd-card-operation-and-recovery.md
Evidence: work/evidence/representative-ab-usb-publication-reader-verification-20260811.md
Decision Evidence: work/evidence/hpt-bounded-export-preimplementation-baseline-and-budget-proposal-20260814.md
Software Evidence: work/evidence/hpt-bounded-export-software-gate-20260814.md
Related: work/tickets/ticket-eliminate-hpt2-exact-read-scaling.md

## Problem

A bounded HPT customer export scans every retained record from the snapshot's
oldest sequence through its end before staging the records inside the selected
wall-clock range. Export latency therefore scales with total retained history,
not primarily with the requested range or published record count.

Representative 2026-08-11 target evidence makes the behavior measurable:

| Controller | Requested range | Retained records scanned | Records selected | Scan time | USB time |
| --- | --- | ---: | ---: | ---: | ---: |
| A | 559 seconds | 1,064,047 | 9 | 1,177.33 s | 0.87 s |
| B | 554 seconds | 623,996 | 10 | 693.90 s | 0.16 s |

Both controllers scanned approximately 900 records per second. The close
agreement and sub-second USB phases show that the 12--20 minute wait is the
full-history HPT selection scan, not USB media throughput or a controller/card
failure. Qualification testing made the histories large enough to expose the
scaling, but the product is intended for continuous 24/7 operation and can
reach similarly large histories in normal service.

This issue is separate from
`work/tickets/ticket-eliminate-hpt2-exact-read-scaling.md`. That ticket removed
active-segment occupancy scaling from each exact read. The current
`serviceHptCustomerExport()` still deliberately starts
`HptTripleCircularIteratorBegin()` at the oldest retained sequence and calls
`HptTripleCircularIteratorNext()` until snapshot end so it can select records,
establish retained bounds, and detect sequence gaps.

## Expected Behavior

A small customer-requested time range must complete within a defined and
communicated product latency budget that does not grow linearly with unrelated
retained history. The implementation must locate or index the relevant range
without weakening snapshot consistency, range-retention checks, gap detection,
three-lane voting, degraded-record reporting, export-capacity enforcement, or
fail-closed media handling.

The UI must continue to show truthful progress and allow a long operation to
be distinguished from a stalled or failed export. USB publication remains a
separate final phase and must begin only after a complete, validated snapshot
selection has been staged.

## Acceptance Criteria

- [x] Establish and owner-approve a quantitative customer latency budget for
      bounded exports; evaluate an initial target of at most 60 seconds for a
      ten-minute range against a snapshot of at least 1,000,000 records.
- [x] Record a before/after complexity and I/O profile that separates range
      location, HPT reads, validation, row staging, and USB publication.
- [x] Avoid walking every unrelated retained record for a small requested
      range; runtime and HPT reads must be bounded by the lookup/index cost and
      the records needed to prove the selected range.
- [x] Preserve exact inclusion and ordering for duplicate, forward-jumped,
      backward-adjusted, and boundary timestamps; do not assume timestamps are
      monotonic unless a separately reviewed product invariant proves it.
- [x] Preserve fail-closed behavior for missing sequences, malformed records,
      ambiguous degraded copies, retired lanes, wrap/reclaim, concurrent
      append, media-generation change, and a requested range no longer fully
      retained.
- [x] Preserve the configured export-capacity rejection before any USB
      temporary file is created or written.
- [x] Retain request-specific telemetry for scanned/selected records, HPT read
      commands/blocks, bulk reads, elapsed selection time, publication phase,
      and terminal reason.
- [x] Pass deterministic model/static/customer-export tests, canonical managed
      Debug and Release rebuilds, package/hash validation, and representative
      Controller A/B target regression without powered SD-card removal.
- [x] Verify small and near-capacity ordinary-history ranges with exact CSV
      content, gap/degradation metadata, and independent USB-reader validation;
      retain passing deterministic wrapped-history coverage and defer the
      destructive physical wrap case to later qualification by owner decision.

## Verification

Use the current 2026-08-11 A/B measurements as the before baseline. Test the
same frozen or explicitly successor-qualified HPT histories with request
ranges before, near, and after the newest retained records. Compare selected
sequence/timestamp sets and output hashes against a trusted full-scan oracle.
Record target elapsed/read telemetry and independently verify the final CSV,
FAT consistency, completion trailer, and absence of temporary files.

Do not close this ticket merely because a test fixture with few records is
fast. Closure requires large-history target evidence on both controller
hardware variants and no newer contradictory incident.

## 2026-08-14 pre-implementation decision gate

The current path and target telemetry are now separated into range location,
HPT read/validation, staging, and USB phases. There is no current range lookup:
Controllers A and B scanned 1,064,047 and 623,996 records for 9- and 10-record
ranges, taking 1,177.33 and 693.90 seconds before sub-second USB publication.
The existing bulk-16 transfer is already at the measured diminishing-return
point, and even a 142,560-record suffix projects to about 158 seconds.

The reviewed correction direction is an explicitly versioned, checksummed
sealed-segment time envelope plus a bounded in-memory range directory. It
retains three-lane voting and exact record validation and does not assume
timestamp monotonicity. Existing media without the feature must follow an
explicit reinitialization/successor-qualification path rather than a silent
metadata migration.

The proposed acceptance budget is terminal `PUBLISHED / COMPLETE` within 60.0
seconds for a ten-minute range against at least 1,000,000 retained records on
both hardware variants, with lookup/validation/staging within 30.0 seconds and
no more than 8,192 records scanned for ordinary and bounded timestamp-adjustment
fixtures. Near-capacity work may scale with selected rows but not with unrelated
history. Owner approval or replacement limits are required before code changes.
See the decision evidence for the full invariant map and alternatives.

The owner approved this budget in the active campaign thread on 2026-08-14.
That approval authorizes implementation and software/build verification. It
does not authorize controller programming or physical target execution.

## 2026-08-14 software gate

The bounded range-envelope correction, deterministic oracle/recovery coverage,
all 27 repository check scripts, all 10 Python model suites, final canonical
managed Debug and Release rebuilds, exact 304-byte telemetry ABI, RAM/ROM
review, and temporary reviewed-source package/hash validation pass. The
accepted 32 KiB FileX publication path is unchanged. See
`work/evidence/hpt-bounded-export-software-gate-20260814.md` for exact source,
artifact, bundle-manifest hashes, complexity, and memory figures.

The two hardware-dependent acceptance items remain open: measured before/after
phase and I/O results on both controller variants, and exact small/near-capacity
CSV/FAT/publication proof across ordinary and wrapped histories. Currently
installed media predates the durable range-envelope feature, so its destructive
reinitialization or successor disposition requires exact operator authorization
before target execution.

## 2026-08-17 A/B million-history target evidence

A temporary append-batched fixture populated both installed exact-CID cards to
1,000,000 retained records in approximately 30 minutes without SD-card removal,
replacement, or reinitialization. After restoring and independently verifying
the reviewed Release candidate, full three-lane readback passed with 1,000,007
healthy records on A and 1,000,009 on B, zero degraded records, and zero gaps.

The bounded ten-minute case completed `PUBLISHED / COMPLETE` in 4.80 seconds on
A and 5.03 seconds on B, scanning 2,876 and 2,875 records and selecting the same
11 inclusive minute records. This is approximately 245 times and 138 times
faster than the respective baselines and passes the approved 60-second and
8,192-record limits. Near-visible-capacity publication also passed on both
controllers, selecting approximately 142,560 rows in 200.01 and 224.33 seconds.

The boundary case exposed an adjacent stale retained-start cache when the
visible floor advanced without changing the physical oldest sequence. The
correction refreshes the cache from
`HptTripleCircularVisibleFirstSequence()`. Its focused checks, canonical managed
Debug/Release rebuilds, and package/hash validation pass; exact successor A/B
target verification is the next protected step. Independent USB-reader proof
and an ordinary wrapped-history case remain open, so this ticket is not closed.

The exact successor Release S-record
`75F108D5E2E77031E107B2D71852545C9FD246B128085CCB7D4F6E256BC37462`
was then programmed and verified on A and B. Independent 4 MiB flash captures
both matched the expanded-image SHA-256
`6EEF5DB83CE2AE71AEB63D598ECAD1CFA7D68475EBFE052301F4FC2A0F97DFC4`.
Same-run boot identity and health passed on both controllers. With all three
physical lane-oldest sequences fixed at 1, the cached visible retained-oldest
sequence advanced by one on each controller after the next production append.
This is the exact moving-retention-floor condition that previously remained
stale. Both graph caches reached ready state, and no SD media was moved or
reinitialized.

## 2026-08-17 independent USB-reader closeout

The Raspberry Pi identified `USB07_32G` as `/dev/sda1`, UUID `4693-0F61`, and
`USB05_4G` as `/dev/sdb1`, UUID `362E-8170`. Both were unmounted for
non-correcting `fsck.fat -n -v`, mounted read-only for content inspection, and
unmounted afterward.

Every campaign CSV passed independent parsing. The credited A/B ten-minute
files each contain exactly 11 healthy sequential records and the credited
near-capacity files contain 142,559 and 142,560 healthy sequential records.
All have zero degraded records, zero snapshot/requested-range gaps, complete
snapshot validation, and a matching `COMPLETE` publication trailer. The two
zero-row boundary artifacts are also internally complete but retain no
populated-range credit. The B retry contains its expected 142,553 records.
No temporary publication artifact exists.

Both FAT copies are byte-identical on each drive. `USB05_4G` has 5,940
allocated clusters with an exact free-space summary and retains the known
root-only label convention. `USB07_32G` has 993 allocated clusters and retains
the previously accepted two-cluster primary-FSInfo offset: primary 1,892,674
free versus 1,892,672 by FAT scan. The check made no change and this drive is
not called clean or repaired. No relevant insertion-time kernel error was
found.

The remaining wrapped-history target case cannot be run in place without
destroying these test sets: `RunWrapDiagnostic` invokes
`HptLogQualificationFormatWrapDiagnostic()` before constructing its bounded
wrapped fixture. Preserve the million-record histories pending an explicit
owner disposition or separate expendable-card authorization.

## 2026-08-17 owner acceptance disposition

The owner accepted the recommended closeout: preserve both installed
million-record histories, accept the passing deterministic wrapped-history
model/invariant evidence for this export-performance ticket, and defer the
destructive physical wrap diagnostic to later qualification. The deferral does
not convert model evidence into target evidence and does not weaken production
gap, reclaim, generation, or fail-closed behavior. With that evidence boundary,
all revised ticket acceptance criteria are directly satisfied and the ticket
is complete.
