# Spike: HPT2 service-life retrieval performance requirements

Status: complete
Type: spike
Updated: 2026-08-26
Next Action: review and approve or revise the proposed HPT2 Qualification Program Roadmap; do not create campaigns yet
Project Map: work/maps/map-smi-firmware-stabilization-and-qualification.md
Campaign: standalone
Campaign Reason: This is a requirements and discovery capture only; it must not change the current owner campaign queue.
Disposition: planned
Produces: work/maps/map-hpt2-qualification.md and work/roadmaps/roadmap-hpt2-qualification.md

## Purpose And Authority Boundary

Preserve the owner's expectations and the current technical findings for a
future effort to qualify HPT2 data logging, retrieval, graphing, export, wrap,
and recovery performance over a controller service life. This concern is
separate from deciding whether an individual SD-card specimen is qualified.

The owner names the resulting effort the **HPT2 Qualification Project**. Its
planning hierarchy begins at `work/maps/map-hpt2-qualification.md` and the
associated Program Roadmap. Campaign creation remains a separate later
approval.

This artifact is the durable input for a later Tool Shed planning exercise. It
does **not**:

- approve an architecture, implementation, performance budget, or
  qualification procedure;
- create an implementation plan, roadmap revision, campaign, workpackage, or
  delivery commitment;
- alter, remove, supersede, or reconcile the current project map, Program
  Roadmap, campaign queue, media contract, or qualification policy;
- authorize firmware changes, builds, controller programming, destructive
  media tests, deployment, or publication; or
- claim that current model tests prove maximum-geometry physical-media
  performance or physical NAND wear.

When the owner explicitly resumes this subject, review this snapshot against
the then-current firmware and project truth before using it to create the full
spike or implementation plan.

## Owner-Stated Outcome And Scope

- Establish that HPT2 can meet data-logging, charting, and export needs over
  years-equivalent use, not merely that an SD card passes an initial
  qualification.
- Keep production logging reliable and responsive while the retained log grows,
  lanes fill, the first wrap occurs, and storage is repeatedly reclaimed and
  reused.
- Make performance include fast retrieval for both charts and exports. A system
  that records quickly but makes customer data slow to retrieve is not
  successful.
- The controller graph exposes only the newest 30 days of data. Customer export
  is limited to the newest 90 days. Years-equivalent retained data are an
  internal storage-aging and wrap stress condition, not authorization to expose
  older customer data.
- HPT2 intentionally appends through each lane and, when the lane is full,
  reuses storage from the beginning. The future work must verify that this
  circular behavior distributes logical writes and remains correct and fast.
  Because an SD card's flash translation layer is opaque, firmware evidence may
  characterize logical block writes and write amplification but must not claim
  uniform physical NAND wear.
- Treat this as its own future planning subject. Preserve its relationship to
  SD qualification, but do not fold it silently into the card-specimen service
  code work.

## Current Firmware Alignment And Baseline

Inspection on 2026-08-26 found that the current firmware is substantially
aligned with the intended bounded customer-access model, but maximum-depth
service-life performance remains unproven.

### Aligned behavior

- The active implementation uses `src/HptTripleCircular.c` with
  `src/Thread_SdCard_entry.c`.
- The configured retention geometry requires 129,600 periodic records and at
  least 142,560 records per lane, leaving explicit capacity for event records.
- Lane progression is sequential and wraps modulo usable segment count while
  skipping retired segments. Stale commit state is invalidated before a reused
  segment receives a new header.
- Customer graph collection is bounded to the newest 30-day suffix, with an
  event reserve, rather than intentionally presenting all retained history.
- Customer export range is clamped to the newest 90 days.
- The range planner maintains bounded per-segment timestamp envelopes and proof
  reads to avoid full record-history scans for bounded exports.
- Foreground logging is serviced ahead of chart, export, and test work in the
  SD-card thread.

### Existing evidence

The following deterministic and source-policy checks passed during this
discovery review:

- `script/test-hpt-triple-circular-model.py`, including more than three wraps,
  bounded exact reads, and durable range envelopes;
- `script/test-hpt-customer-export-model.py`, including a million-record
  bounded lookup;
- `script/test-hpt-graph-cache-model.py`;
- `script/check-export-range-policy.ps1`;
- `script/check-hpt-customer-export.ps1` with 40 of 40 checks; and
- `script/check-hpt-commissioning.ps1` with 100 of 100 checks.

Existing recorded baselines include approximately 4.80 and 5.03 seconds for
the two million-record bounded-export model cases. Existing near-capacity
90-day export evidence for approximately 142,560 rows is about 200 to 224
seconds. These are observations, not owner-approved future acceptance budgets.

### Remaining scaling and proof concerns

- `HptTripleCircularIteratorBegin()` begins at the oldest retained sequence.
  Range positioning avoids scanning every preceding record, but it still walks
  preceding segment headers before reaching a later indexed position. Initial
  seek work therefore remains proportional to the number of segments.
- Store open/recovery reconstructs oldest, newest, active, and range-index state
  by reading every non-retired segment header. At the supported 32 GiB maximum,
  the derived upper bound is approximately 10,922 segments per lane and 65,532
  header-sector reads across six lane copies. Target duration at this depth has
  not been measured.
- Deterministic models exercise repeated wraps, but the first and repeated
  wraps have not yet been qualified at maximum depth on physical target media
  under normal concurrent logging and recovery conditions.
- Passing the current million-record export model does not prove bounded graph,
  export, reopen, and recovery performance through maximum geometry, reclaim,
  degradation, and reboot.
- The graph suffix's event reserve must be challenged with realistic dense-event
  histories rather than assumed sufficient from minute-record geometry alone.

## Requirements For The Future Discovery And Plan

### Define separate, quantitative performance budgets

The future effort must propose and obtain owner approval for distinct budgets;
it must not hide all responsiveness behind a single end-to-end number. Measure
at least:

- media open, classification, and normal recovery after boot;
- full-scan fail-closed recovery when fast recovery state is unavailable;
- graph-cache reconstruction;
- graph initial open, zoom, pan, and live refresh for the newest 30 days;
- bounded ten-minute export;
- full 30-day and full 90-day export;
- production minute-record and event-record append latency, including worst
  observed delay while graphing or exporting;
- export and graph cancellation latency; and
- time to return to a known usable state after interruption or failure.

Retain the existing controlled ten-minute export requirement as an input where
applicable, but do not infer unapproved budgets for the other surfaces from it.

### Measure work as well as wall-clock time

Elapsed time alone can conceal geometry-dependent growth. Evidence should also
record, where technically practical:

- segment headers visited;
- records scanned, selected, decoded, and emitted;
- SD commands, blocks read and written, retries, and persistent failures;
- range-index entries, candidate segments, envelope proofs, and fallback paths;
- writes and flushes per appended record;
- maximum append delay while readers are active;
- controller, HMI, communications, alarm, and watchdog progress; and
- cancellation checkpoints and remaining work.

The analysis must identify expected complexity and demonstrate that the
measured work remains bounded by the customer request or an approved recovery
case instead of silently scaling with years of retained history.

### Exercise representative geometries and service-life states

At minimum, cover the smallest and largest supported HPT2 geometries at these
states:

- empty and newly commissioned;
- partially populated;
- segment transition;
- near lane capacity;
- first reclaim and first wrap;
- multiple completed wraps;
- one or more retired segments;
- degraded one-copy and two-copy read conditions; and
- reboot or reopen at each important state.

The workload must include realistic mixtures of minute records, door events,
alarms, warnings, power events, configuration events, duplicate timestamps,
forward clock changes, backward clock changes, and event bursts dense enough to
challenge the graph event reserve.

Use reproducible workload profiles and seeds. Include both accelerated
years-equivalent generation and selected target-duration or physical-media
runs so model speed is not mistaken for controller/media performance.

### Preserve correctness while measuring speed

Performance changes and budgets must not weaken:

- record ordering and content;
- mirror quorum, degraded-mode handling, and provenance;
- protected-record retention across reclaim and wrap;
- the 30-day graph and maximum 90-day customer-access limits;
- exact CSV range, row count, health fields, and completion trailer;
- consistent read snapshots while new records are appended;
- fail-closed behavior for proven media, filesystem, or transport failures; or
- publication rules that prevent a partial USB file from appearing finalized.

Export evidence should include an independent USB-reader check of file content
and hashes when a USB artifact is part of the run.

### Verify concurrency and controller responsiveness

Run graph and export retrieval concurrently with ordinary minute and event
logging. Confirm that control behavior, alarms, HMI, communications, and
watchdog progress remain responsive and that a slow or cancelled reader cannot
starve the production append path. Define how a graph or export establishes a
consistent range while appends continue.

### Exercise failure and recovery paths

Include controlled interruptions during append commit, segment transition,
range-envelope update, reclaim, graph generation, and export publication. Also
exercise corrupt or stale recovery metadata, read retry and persistent read
failure, USB write/publication failure, cancellation, and whole-controller
power interruption where qualified.

SD-card insertion, removal, or replacement must occur only with controller
power off. Powered SD-card hot swapping is prohibited and is not a valid
service-life or failure-injection technique.

### Characterize logical write distribution

Collect logical block-address distribution, writes and flushes per record,
segment-header and retirement-metadata frequency, metadata hot spots, and
estimated firmware-visible write amplification across multiple wraps. State
clearly that these measurements show firmware logical-write behavior, not the
SD card's undisclosed physical flash placement or remaining NAND life.

## Candidate Mitigations To Evaluate

These are recommendations for the future spike, not approved changes:

1. Evaluate direct iterator positioning from the range index so a bounded graph
   or export does not first walk segment headers from the oldest retained
   segment.
2. Evaluate a redundant, checksummed recovery directory or checkpoint that can
   restore normal open state without reading every segment header. Keep a full
   media scan as the fail-closed recovery fallback when the checkpoint is
   absent, stale, inconsistent, or corrupt.
3. Instrument algorithmic work and append interference in firmware so target
   evidence can distinguish media latency from avoidable linear traversal.
4. Separate quick host/model invariant checks from long target profiles. Run
   the fast suite for relevant changes and reserve maximum-depth physical-media
   profiles for HPT/media-format changes and release qualification.
5. Establish explicit graph, export, recovery, and append budgets before
   selecting an optimization, then compare candidate designs against the same
   correctness and recovery cases.

## Evidence And Reproducibility Expectations

Each formal future run should identify:

- firmware version and hash, media-format version, and controller variant;
- SD protocol state, CID validity, representative-sector-read state,
  manufacturer name and MID, OEM/application ID, product name, product
  revision, serial number, manufacturing date, and complete raw 128-bit CID,
  captured at baseline and after any relevant replacement, power-off
  reinsertion, initialization, or reboot;
- SD capacity and derived HPT2 geometry;
- workload profile, seed, starting lane/generation/wrap state, record counts,
  event mix, and timestamp range;
- requested graph/export range and actual selected range;
- all applicable elapsed-time and algorithmic-work measurements;
- degraded state, injected fault, retry, fallback, and recovery details;
- raw telemetry, output hashes, and independent validation results; and
- the canonical USB label and USB evidence state when output is written to USB.

### USB-absence override boundary

The related service-code direction allows an operator to run tests without a
USB drive under an explicit override. USB absence under that override must not
become an HPT2 or SD test error. Keep technical outcome separate from evidence
state, such as `TEST PASS` and `USB EVIDENCE NOT RECORDED - OPERATOR OVERRIDE`.

For this future service-life effort, however, a run without the durable evidence
required by the approved profile must not satisfy a formal performance or
qualification evidence gate unless an explicitly approved equivalent evidence
path captured the complete result. The ability to execute is not the same as
the authority to grant qualification credit.

## Open Decisions

- Owner-approved latency, throughput, cancellation, recovery, and append-jitter
  budgets for each performance surface.
- Exact minimum and maximum supported geometry cases and physical SD specimen
  matrix.
- Required duration and number of physical wraps versus accelerated/model
  coverage.
- Approved workload distributions, event-burst bounds, and clock-change cases.
- Whether direct indexed seek, a recovery checkpoint, both, or another design
  is justified after baseline measurement.
- Recovery-checkpoint format, redundancy, update frequency, integrity rules,
  compatibility, and full-scan fallback behavior if that design is selected.
- Instrumentation retained in production versus qualification-only builds.
- Required controller/HMI responsiveness observations and watchdog margins.
- What evidence equivalence, if any, can give formal credit to a run executed
  under the USB-absence override.
- Which roadmap, plan, workpackage, qualification, and release surfaces should
  be created or revised when the owner authorizes planning.

## Exit Deliverables For The Later Full Spike

Before a later implementation plan is considered ready, the future spike should
produce:

- approved performance and correctness requirements;
- reproducible current-firmware baselines across the selected geometry and
  lifecycle matrix;
- scaling and bottleneck analysis that explains both elapsed time and work;
- documented gaps, failure behavior, and residual risks;
- comparison of direct-seek, recovery-index/checkpoint, and any other viable
  alternatives;
- a recommended architecture with compatibility and recovery rationale;
- proposed fast-regression and long-target qualification profiles; and
- explicit inputs, dependencies, evidence gates, and owner decisions for the
  eventual Tool Shed implementation plan.

The future spike must not close solely because the existing million-record
export model is fast. It must establish bounded and correct graph, export,
reopen, and recovery behavior through the approved maximum geometry, wrap,
reclaim, degradation, concurrency, and reboot cases.

## Related Current Artifacts

- `work/spikes/spike-firmware-service-code-sd-qualification-direction.md`
  captures the separate proposed firmware service-code, operator-pause,
  power-off, USB-output, and USB-absence-override direction.
- `docs/hpt-triple-circular-media-contract.md` defines current HPT2 media
  behavior and invariants.
- `docs/sd-card-operation-and-recovery.md` defines current SD operation and
  recovery behavior.
- `work/tickets/ticket-eliminate-full-history-scan-latency-for-bounded-hpt-customer-exports.md`
  records the completed bounded-export change.
- `work/tickets/ticket-eliminate-hpt2-exact-read-scaling.md` records the
  completed exact-read scaling work.
- `work/wp/completed/wp-optimize-hpt-triple-mirror-read-performance.md` and
  `work/wp/completed/wp-correct-hpt-graph-rolling-cache-refresh.md` preserve
  related completed implementation history.
- `work/wp/active/wp-triple-mirrored-circular-hpt-log-store.md` remains current
  project context and is not modified or superseded by this spike.

## Recommendation

Keep this artifact as a documented requirements snapshot until the owner is
ready to commission the separate HPT2 service-life spike and full Tool Shed
plan. At that time, confirm the owner-stated scope, define quantitative budgets,
measure the maximum-depth baseline, and evaluate direct seek and fast recovery
state before committing to an implementation architecture.

## Follow-Up

- [ ] Owner explicitly reopens HPT2 service-life performance discovery and
  authorizes creation of the full spike or plan.
- [ ] Reconcile this snapshot against then-current firmware, media contract,
  qualification evidence, and project state.
- [ ] Obtain owner decisions for quantitative performance budgets and the
  target qualification matrix before planning implementation.
