Implementation Walkthrough — eCRF cut, two traces

Trace 1 — MedDRA auto-coding

1.0 The input on disk

ae.csv as DM delivers it. AECOM is present but is the PHI trace's column — the coding trace never reads it. The code columns are absent because deriving them is this trace's job.

Tier 2 indexes the MedDRA release assembled from the licensed ASCII conversions, the first artifact it needs on disk:

merged on currency = 'Y', joined LLT→PT on pt_code, PT→HLT on hlt_code, HLT→HLGT on hlgt_code, HLGT→SOC on soc_code; each term is indexed under llt_name + "[PT: " + pt_name + "]". The full chain is LLT → PT → HLT → HLGT → SOC: related preferred terms are grouped into High Level Terms by anatomy, pathology, physiology, etiology or function; related HLTs in turn link to High Level Group Terms; and HLGTs group into System Organ Classes (by etiology, manifestation site or purpose).

1.1 · Tier 1 — eCRF intake

Keys ae.csv on STUDYID/USUBJID/AESEQ, copies AETERM through byte-for-byte, withholds AECOM from this trace. No columns added — the verbatim reaches Tier 2 exactly as typed, because Tier 2 measures closeness to what the site wrote. (The spatial/PDF front-end is not invoked: the input is tabular.)

1.2 · Tier 2 — Cactus Needle3: embed, index, retrieve

The verbatim, lower-cased (Needle3's embedding is case-sensitive), is encoded to a 3072-dim unit vector; because stored vectors are L2-normalized, the index's inner-product distance is cosine, so the cheap ip search returns exact angular closeness. The active term vectors sit in a layered HNSW graph; a query enters at the top entry point, follows ip links greedily while the score improves, steps down at each local maximum, and at layer 0 runs a bounded priority-queue search to return the top-K. Index parameters: M = 32, efConstruction = 200, efSearch = 64. Measured costs [RM]: embed 31 ms + search 0.5 ms per verbatim over a 3,371-term synthetic dictionary (index resident 40.4 MB, recall@10 0.994 against exact search). At a full release's ~86,800 terms, measured on random vectors of the same width (index cost only, not retrieval quality): search 5.8 ms, resident 1,040 MB (1,017 MB vectors + 23 MB graph). On the real path Tier 2 hands its top 10 to the re-ranker; the tables show the three exported after re-ranking.

Zoomed on AESEQ 1, "severe throbbing pain in back of head" — top-3 [RM] (each candidate carries its full hierarchy path):

Zoomed on AESEQ 2, "feeling lightheaded when standing up" — top-1 [RM]: Lightheadedness → Dizziness postural (99013575), cosine 0.9582.

Ranks 1–2 of AESEQ 1 share a preferred term but are distinct lowest-level terms; the re-ranker resolves them to the same PT/code, so the write-back is unambiguous whichever LLT ranks first. Tier 2 has appended LLT, PT, PT_CODE, SOC, cosine and stopped — it proposes and measures, it decides nothing.

Needle3's cosines are compressed: every candidate in this trace, the unrelated QT term included, scores between 0.93 and 0.97. A cosine on its own separates little, and no fixed cosine cutoff carries over from one dictionary to another; ordering and gating are the re-ranker's job.

1.3 · Tier 3 — Laya: what it adds to a coding row

Tier 4 scores each candidate from a feature row built out of upstream columns, never out of the raw string:

x_k = [ cosine(Tier 2),
        LevenshteinSim(Tier 4) = 1 - EditDistance(s,m)/max(len s, len m),
        JaroWinkler(Tier 4),
        I(SOC match)(Tier 4),
        retrieval rank(Tier 2),
        + P(unambiguous)(Tier 3)   # real path only; the stand-in has no Tier-3 scalar
      ]
y_k = XGBoost(x_k)            # real: binary:logistic booster, 164 trees
                              # stand-in: fixed weights on the first four, summing to 1

Laya supplies P(unambiguous): the probability that the verbatim reports one specific problem the participant actually had. That is what lets a row whose term matches well but whose sentence negates or blurs it be pushed below the gate. Measured [RM] on the four worked verbatims, all plain reports: 0.911 (AESEQ 1), 0.872 (AESEQ 2), 0.919 (AESEQ 3), 0.861 (AESEQ 4). On the held-out set it separates negated from plain verbatims only moderately (AUROC 0.74), and no negated verbatim was auto-coded there. The stand-in does not emit the scalar, so the [SI] trace runs its fixed-weight composite without it. Either way the gate is a single threshold on y_k.

1.4 · Tier 4 — XGBoost: re-rank and gate

The re-ranker is trained once (here on synthetic data) and is not re-derived per record, so the trace reports its score, not its trees. This run sets the gate to 0.90; the build's slider default is 0.88. On the real path the trained re-ranker scores all four worked verbatims far below either value, so all four go to the human coding queue at 0.88 and at 0.90 alike, each with the right preferred term ranked first. That is the measured behavior for paraphrased verbatims: on the held-out set, colloquial verbatims reached the right term at rank 1 about half the time and none cleared the gate (EVALUATION.md, Stage 3).

The stand-in computes the same four lexical feature columns per candidate (cosine via hashed bag-of-words, Levenshtein-sim, Jaro-Winkler, SOC-match) and a fixed composite with Σw = 1. Its [SI] scores are on a different scale from the trained re-ranker's, so the two columns are not comparable number for number; at 0.90 the stand-in also flags all four. Where the real path does what a lexical matcher cannot is retrieval: it ranks Dizziness postural first for "feeling lightheaded when standing up", which the stand-in codes as Headache, and Electrocardiogram QT prolonged first for "marked prolonged heart rhythm interval". What it does not yet do is score those right answers high enough to auto-code them.

1.5 · Write-back and export

ae_coded.csv carries the top three candidates per verbatim. Rank 1 has status AUTO_COMMIT_TO_SDTM (the coded fields are committed) or FLAGGED_FOR_HUMAN_CODER (they are a proposal for the coder to accept or override); ranks 2–3 are ALT_CANDIDATE. Rank-1 rows written on the real path in this run [RM]:

AECOM is untouched in this file — scrubbing it is the PHI trace's output, a separate artifact. The exported rows also carry HLT and HLGT as evidence columns next to each candidate so a reviewer sees the full path (LLT → PT → HLT → HLGT → SOC) beside every proposal; they are not standard SDTM variables — the write-back fields remain AEDECOD/AEPTCD/AESOC, and Tier 4 never scores on the hierarchy levels.

1.6 · App tie-in (coding)

The MedDRA write-back screen renders 1.5 as an editable table: each AUTO_COMMIT_TO_SDTM row shows its top PT — with its HLT · HLGT · SOC path alongside — and score for accept; each FLAGGED row shows the pre-populated candidates for the coder to pick or override, and an override writes a reason to the audit. The reviewer sets the gate here (default 0.88) and the effective value goes into the audit with the run. The pre-flight panel prints the hierarchy chain the dictionary actually carries (LLT > PT > HLT > HLGT > SOC, or a degraded subset if the source lacked those levels), and that chain is written to the audit with every run.

Trace 2 — PII/PHI detection and redaction

2.0 · The free-text on disk, across domains

These are the boxes a coordinator types into. The cut deliberately contains four cases: a direct leak, a look-alike trap, a mosaic, and a clean negative — all in eCRF fields, no page.

The entity schema is the HIPAA safe-harbour set (Name, DOB, Phone, MRN, Address, …). Every non-numeric column in each domain is scanned — content-based discovery, not a hard-coded AECOM list or a name-suffix rule — which is what makes the scan both domain-agnostic and gap-free.

2.1 · Tier 1 — intake + content-based discovery

Discovery is content-based: every column that is not strictly numeric (text, dates, coded values, flags) is scanned; only columns typed as a number are skipped. A pre-specified list of column names was removed because it is guaranteed to miss PHI hiding in any column whose header did not match — and missing PHI is exactly the failure this program exists to prevent. Applied per domain on the sample cut:

The operator confirms or deselects columns in the app's pre-flight before compute — so a known-safe structured column (a date you don't want touched) can be excluded interactively. A missing value stays missing, not an empty string that would scan silently clean. No page, no layout, no coordinate map — the input is tabular.

2.2 · Tier 2 — Cactus Needle3: typed span extraction

The extractor is schema-typed, not a regex library; the stand-in implements the declaration as explicit patterns, shown verbatim so a reviewer can audit exactly what matches — phone \(?[0-9]{3}\)?[-. ]?[0-9]{3}[-. ]?[0-9]{4}, MRN \b[0-9]{3}-[0-9]{3}-[0-9]{2}\b, DOB \b[0-9]{2}/[0-9]{2}/[0-9]{4}\b, postcode (?<!\d)[0-9]{5}(?!\d), initials \b[A-Z]\.[A-Z](?:\.(?=[^0-9a-zA-Z]|$))?, and a patient name only when introduced by a title keyword (Patient|Mr|Ms|Mrs|Dr) or in surname-comma form. The title-keyword guard is what stops a bare capitalized clinical word from being read as a name. Spans carry in-field character offsets; there is no page and no bounding box.

The offsets below are the exact output of the shipped stand-in on sample_data/ (verified 2026-10-13):

The two italicised none rows are the look-alike test the typed schema exists to pass: a mid-sentence Alanine matches no name pattern (the title-keyword guard) and 120 mg/dL matches no MRN/DOB/ZIP pattern, so both survive while the patient name beside them is flagged. The PR row mixes direct (MRN, DOB) and indirect (initials, postcode) attributes — that is the mosaic case Tier 3 must score.

On the real path Tier 2 is the union of Needle3's typed extraction and the same patterns; a range both proposed is marked both. Its proposals on these fields [RM]: Jane Miller and the phone on AE row 1, Jane Miller on AE row 2, and the PR row's initials, DOB and postcode were found by both; the PR row's MRN by the pattern alone. Needle3 also proposed four look-alikes the patterns do not: 10:15 AM and 50 (from Alanine 50 mg) as dates of birth, 120 (from 120 mg/dL) as a ZIP, and Sinus rhythm on the EG row as a person name. Tier 4 rejects all four (§2.4). Needle3's extraction alone is not a redactor: on the held-out set its precision was 0.42.

2.3 · Tier 3 — Laya: calibrated field-level probabilities

One non-generative pass per field returns the probability that direct identifiers are present and the probability that indirect attributes jointly re-identify. P(Quasi) is the joint re-identification probability over indirect attributes; it should rise only where two or more co-occur, which in this cut is the PR row alone.

The single-name rows keep P(Quasi) low because no indirect set co-occurs; the PR row raises it because initials and postcode appear together. That joint risk is a field-level judgment Tier 2 cannot produce — Tier 2 flagged the MRN and DOB as direct spans but said nothing about the combination of the indirect attributes, which is precisely Tier 3's output.

Measured, P(Quasi) behaves as designed. P(direct) does not: it is 0.67 on AE row 1 and close to zero on AE row 2 and the PR row, though all three hold direct identifiers. That matches the held-out evaluation, where Laya's direct-identifier answer was weak and biased low (AUROC 0.69). The redactions in §2.4 are right anyway, because the trained span gate leans on Tier 2's span evidence more than on P(direct).

(Stand-in note: the stand-in reports P(direct) = 0.99 whenever a direct span exists and P(quasi) = 0.96 when two or more indirect spans co-occur, none when there is no indirect span. The [RM] values are Laya's field-level answers, one per question, not per entity.)

2.4 · Tier 4 — XGBoost: per-span redaction and scrub

Per span the head predicts a redaction probability from the Tier-2 span match, the two Tier-3 probabilities, and in-field cues (relative character position, digit ratio, capitalization, a preceding label word). The full per-token feature set also includes page geometry — bounding box and normalized vertical position — which the eCRF path has no page to supply, so in-field position and lexical cues stand in for it and the score reduces to the span-match and direct-identifier test: P(Redact) = 0.99 when the token falls in a Needle3 span or the field's direct-identifier probability exceeds 0.8, else 0.01; mask above the configured cutoff (default 0.85, a reviewer parameter). The scrub is a string replace on the field, shown as the exact operation so the result is reproducible:

SAN_AECOM = str_replace_all(AECOM, "(?i)Jane Miller", "[REDACTED_PATIENT_NAME]")
SAN_AECOM = str_replace_all(SAN_AECOM,
              "\\(?[0-9]{3}\\)?[-.]?[0-9]{3}[-.]?[0-9]{4}", "[REDACTED_PHONE]")
AEPHI     = if_else(SAN_AECOM != AECOM, "Y", "N")

The 0.99 / 0.01 rule above is the stand-in's. On the real path a trained XGBoost gate scores each Tier-2 candidate from 18 named features (which proposer found it, its entity type, P(direct), P(quasi), the number of indirect spans in the field, digit ratio, capitalization, relative position, length and a preceding label word) and masks above the same 0.85 cutoff.

Resulting sanitized field, AE row 1: CRC Note: Patient [REDACTED_PATIENT_NAME] called site cell phone [REDACTED_PHONE] at 10:15 AM., with AEPHI = "Y"; AE row 2 keeps Alanine 50 mg and 120 mg/dL intact while the patient name beside them is scrubbed. The tag vocabulary ([REDACTED_PATIENT_NAME], [REDACTED_PHONE], …) is a configuration choice and the audit records the scheme in force for the run. Because the input is a database field there is no manifest and no text-layer deletion — those belong to a page, which this build does not process.

2.5 · Exports (PHI)

Two artifacts, neither of which is the coding file. phi_flags.csv, one row per detected span (page and bounding-box columns do not appear, because the input is a field, not a page):

and a sanitized copy of each scanned domain — sanitized/ae.csv, sanitized/pr.csv, sanitized/eg.csv — carrying the original columns plus SAN_<field> and <field>_PHI_FLAG per scanned column. The audit row for this trace records the selected-column list, the content-based discovery mode and the tag scheme in force, the redaction cutoff, the per-tier path (trained or stand-in) — so the privacy run is reproducible from its audit alone and shares no audit line with the coding run.

2.6 · App tie-in (PHI)

The PHI sanitization screen renders 2.4 as a raw-vs-sanitized inspector per row with an Approve-Redaction action that logs reviewer and time, and shows the 2.1 discovery table so a reviewer can confirm coverage before trusting the scrub. The reviewer confirms or deselects scan columns and sets the cutoff here; those effective choices go into the audit with the run. End users write no code.

Trace legend (properties of this build)

• [RM] = measured output of the named model versions on the named hardware and date: Needle3 (cactus-needle 3.2.8), Laya 0.4.13 (convaiinnovations/laya@55cf4c4e, temperatures fitted on synthetic data) and XGBoost 3.4.1 gates trained on synthetic data, on an Intel Xeon @ 2.80GHz, 4 vCPU, no GPU, 2026-10-22 (EVALUATION.md); [SI] = the deterministic stand-in's output on this record; untagged numbers = configured parameters (gate 0.90 in this run, default 0.88; redaction cutoff 0.85; index M/efC/efS = 32/200/64) or dictionary content.
• The stand-in reproduces the contract and the gates exactly. For PHI it reproduces the trace, because identifier detection is pattern-shaped and the eCRF scrub is a string replace — every [SI] offset in §2.2 was verified against a shipped run on 2026-10-13. For coding its retrieval is lexical, so its [SI] scores are not comparable with the [RM] column; with the models dir built by eval/fetch_models.py the same tables fill with [RM] numbers, columns unmoved, as they do in this deck.
• Two parameters are deployment decisions the audit records, not fixed constants: the coding gate (0.88 vs 0.90; on the real path no score in this cut or in the held-out set fell between them) and the redaction tag scheme. Both are reviewer-set in the app and written to the audit with the run.

Verify this walkthrough yourself

Point either app at ../sample_data/ (the worked cut plus a synthetic demo dictionary) and Execute: no code edits, no setup beyond pip install -r requirements.txt. The PHI trace reproduces the [SI] values in §2.2–§2.5 exactly; the coding trace runs both routing branches with honest [SI] scores. For the [RM] values, build a models dir as in ../eval/README.md, set it as Models dir in the sidebar and run again; ECRF_MODELS_DIR=<models dir> python tests/qc_test.py checks that the run reproduces them. What was checked, what bugs were found and fixed, and how to re-run the QC harness (tests/qc_test.py): QC_REPORT.md.