NHANES → ICES Harness Port
Required datasets & 3-million-record batching

Two deliverables for mimic_mortality_harness.html: (A) the exact ICES Data & Analytic Services (DAS) holdings that reproduce — and exceed — the reporting the harness currently derives from the harmonized NHANES 1988–2018 data, and (B) a bounded-memory modification that ingests 3,000,000+ records by incremental batching instead of loading all rows into memory.

AWhich ICES data files you need

The harness consumes seven NHANES variable domains. Each maps to one or more ICES holdings from the DAS third-party-research catalogue. The linkage spine changes from NHANES SEQN to the ICES IKN (encrypted key number); the harness's SEQN-merge logic maps 1:1 onto IKN-merge.

Minimal viable request (runs the harness's mortality + staging + Rx + §08 reporting)

B3-million-record batching modification

The bottleneck (as-shipped)

The stock harness already streams each file, but it then (1) pushes every kept row into an in-memory rows[], (2) merges all persons on SEQN into five maps DATA.nhanes.{demo,cond,mort,lab,rx}, and (3) builds a per-person RESULTS[] in runAll(). Peak memory is O(N persons) — fine at NHANES scale (~10⁵), fatal at 3×10⁶.

The fix — bounded-memory batched fold (harness_batch.js)

Process records in fixed-size batches; fold each person directly into O(#oracles × #stages) running aggregates; then discard the raw rows before the next batch. Peak memory = O(BATCH_SIZE + aggregates), independent of total record count. The fold reuses the harness's own routeNHANES, computeStage and survivalLE/lifeYearsGained on a self-contained person record, so the numbers are identical to the stock path.

Verified by smoke_batch.mjs: 3M synthetic joined rows streamed in 60 batches; peak retained rows never exceeded the batch size; every aggregate (per-oracle counts, life-years, observed deaths, KDIGO/ADA stage breakdown, PHQ-9/LDL non-mortality endpoints) matched an independent single-pass reference exactly; all cells <6 suppressed. A naive all-in-memory pass on 3M rows would hold multiple GB (3M row-objects + five SEQN maps + RESULTS[]) and crash a browser tab.

Public API

HarnessBatch.streamFold(source, {
  batchSize,           // rows held at once (default 50 000)
  keepFn,              // column whitelist (reuse the harness nhanesKeep)
  onBatch(rows),       // fold callback; rows are DISCARDED after it returns
  onProgress(bytes,n)  // UI progress
}) -> {rows, bytes, batches, peakBatchRows}   // peakBatchRows <= batchSize (guaranteed)

HarnessBatch.makeAggregator({route, stage, maxRR, lifeYears})
  .foldPerson(personRecord)   // O(1); no per-person retention
  .aggregateRows()            // §04 rows, <6 suppressed
  .stageRows()                // §04b/§04c stage breakdown, <6 suppressed
  .nonMort()                  // §08 depression + LDL

// source = browser File (uses File.stream()) OR a Node async-iterable of chunks.

Wiring it into mimic_mortality_harness.html (four edits)

1. Add <script src="harness_batch.js"></script> (flat root — respects the bundle's no-subdirectory rule).
2. Add a "Batched (3M+) mode" checkbox + a BATCH_SIZE input + a progress bar to §01. Auto-enable when a single file's .size exceeds a threshold (e.g. 200 MB).
3. In batched mode, replace the streamExtractColumns → ingestNHANES → runAll path with: const agg = HarnessBatch.makeAggregator({route:routeNHANES, stage:stageInline, maxRR:oracleMaxRR, lifeYears:lifeYearsGained}); await HarnessBatch.streamFold(file, {batchSize, keepFn:nhanesKeep, onBatch:rows=>rows.forEach(r=>agg.foldPerson(toPerson(r)))});
4. Point the existing renderers (§04 renderAgg, §04b/§04c, §08) at agg.aggregateRows() / agg.stageRows() / agg.nonMort() instead of iterating DATA.nhanes.demo. §03 per-person rows are already withheld under enclave rules, so nothing there changes.

Multi-file inputs under streaming (the SEQN/IKN-merge question)

A single-pass fold needs each row to be a complete person. Two supported modes:

• Joined mode (recommended for 3M): supply one pre-joined extract — one row per IKN with demographics + condition flags + mortality + the ~15 staging labs + a delimited drug list. Single pass, O(batch) memory. In an ICES extract this join is trivial to produce inside the enclave (SAS/R) before export.
• Two-pass streaming join (multi-file): pass 1 streams every file into a compact Map<IKN, packedRecord> that holds only the ~15 kept fields as a single delimited string (not a JS object per column); pass 2 iterates the map, folds, discards. Peak memory = O(#persons × ~15 small fields) — for 3M that is a few hundred MB (feasible), versus multiple GB for the stock full-object merge. Document the trade-off and prefer joined mode past ~1–2M.

ICES adaptation notes

• Swap SEQN→IKN in nhanesKeep, sid(), and the merge keys (one-line changes).
• Replace self-report routing with cohort membership: routeICES(ikn) reads pre-computed cohort flags (ODD/CHF/COPD/…) carried as columns on the joined extract — so routing stays an O(1) per-row lookup, exactly like routeNHANES.
• Staging labs come from OLIS under LOINC/ICES codes; map them to the harness's canonical NHANES codes (LBXSCR, LBXGH, …) in the extract, so computeStage is unchanged.
• Keep the <6 suppression (already present) — it satisfies ICES cell-size release rules on every exported table.