Case study: four-tier coding and de-identification, traced

Read this first

Two paths, and where each number comes from

Every tier has a labeled deterministic stand-in, tagged [SI] in the app's banner and in every audit row. The coding and de-identification apps also have a real model path, which turns on tier by tier only when a trained artifact in the models dir actually loads; an artifact that is present but cannot be used stops the run instead of falling back to a stand-in. Numbers measured on the real path are tagged [RM] below.

Untagged numbers are configuration (the coding gate, default 0.88 and set to 0.90 in the worked run; the redaction gate 0.85; index settings M 32, efConstruction 200, efSearch 64) or dictionary content.

The design: four tiers, numbers between them

A single text-to-code model folds "the dictionary match was close" and "this token is a drug name, not a patient name" into one set of weights, leaving nothing in between to read or to set a limit on. Here the work is split so each tier writes named columns the next tier reads, and the decision tier receives only numbers derived from the text, never the text itself. A decision then becomes a row of readable columns, and its threshold a parameter a reviewer can move without retraining.

Why separate tiers, and what measurement said about it. Two problems motivate the split. A look-alike (a drug name or a lab value next to a real patient name) defeats a pattern matcher, and two candidates can be equally close in retrieval, so similarity alone cannot pick the code. The argument held up only in part. On coding, Laya's answer plus the re-ranker kept every negated verbatim below the gate on the held-out set. On de-identification, the look-alikes in the worked fields were kept, but because of the span evidence and the trained gate, not because of Tier 3: Laya's direct-identifier probability was weak (AUROC 0.69) and mostly ignored by the gate.

Worked example 1: coding

The synthetic cut (protocol PROTO-2026-X89) opens with four paraphrased verbatims in sample_data/ae.csv. The coding app reads only AETERM; the comment column next to it belongs to the de-identification trace. Every exported row carries the full LLT, PT, HLT, HLGT and SOC path as evidence columns (HLT and HLGT are not SDTM write-back variables; the write-back fields are AEDECOD, AEPTCD and AESOC).

Preferred-term names are the demo dictionary's (MedDRA-style spelling, placeholder codes). The two score columns are on different scales and are not comparable number for number. Routing is the same on both paths and at either gate (0.88 or 0.90).

• Retrieval is where the real path helps. It ranks the right preferred term first for all four, including "Dizziness postural" for "feeling lightheaded when standing up", which the lexical stand-in codes as Headache.
• It does not yet score those right answers high enough to auto-code them. The trained re-ranker scored all four between 0.04 and 0.12, far below either gate, so all four go to a coder at 0.88 and at 0.90 alike. That matches the held-out result for colloquial verbatims (top-1 52%, none auto-coded): the re-ranker was trained on a dictionary where most right matches are lexically close.
• Similarity alone separates little. For AESEQ 1 the real path's top three were Throbbing pain in head (0.9629), Pain in back of head (0.9605) and Electrocardiogram QT prolonged (0.9326): every candidate, the unrelated one included, scored between 0.93 and 0.97 [RM]. Ordering and gating have to be the re-ranker's job.
• Laya read all four as plain reports: P(unambiguous) 0.861 to 0.919 [RM]. The stand-in has no Tier 3 and scores without it.
• The gate does fire on easy cases. On the stand-in path the exact verbatim "headache" auto-commits with a score of 1.0, and across the whole sample cut 50 of 151 verbatims auto-commit at the default gate [SI].

Worked example 2: de-identification

The same cut carries four free-text fields built to test four cases. The app scans every non-numeric column in each domain; identifiers turn up only in AECOM and PRDSC.

Spans and redaction probabilities

Offsets are in-field character positions from the stand-in run; the real path flagged the same seven spans. All seven are above the 0.85 gate on both paths.

• The look-alikes survive on both paths. "Alanine 50 mg" and "120 mg/dL" are kept while the name beside them is scrubbed. The stand-in never proposes them (a name needs a title in front; the number patterns do not match). On the real path Needle3 did propose four look-alikes, 10:15 AM and 50 as dates of birth, 120 as a ZIP code and Sinus rhythm as a person name, and the trained gate rejected all four [RM].
• The mosaic case. The PR note's initials and ZIP code are each harmless alone; the stand-in redacts indirect attributes only when two or more co-occur in a field, which they do here.
• Laya's field probabilities are where an earlier version of Deck B failed most. [RM] P(quasi) behaved as designed: 0.713 on the PR note, 0.404 or lower elsewhere. P(direct) did not: 0.673 on AE 1, but 0.013 on AE 2 and 0.036 on PR 1, though both hold direct identifiers, and 0.012 on the clean EG note. (An earlier version of the deck showed 0.998.) The redactions are right anyway because the gate leans on the span evidence. The stand-in reports P(direct) 0.99 when a direct span exists and 0.05 otherwise, and P(quasi) 0.96 when two or more indirect spans co-occur.

Output. AE row 1 becomes CRC Note: Patient [REDACTED_PATIENT_NAME] called site cell phone [REDACTED_PHONE] at 10:15 AM. with its flag set to Y; the EG note is unchanged with flag N. The tag vocabulary is configuration, recorded in the audit. Because the input is a database field, there is no page, bounding box or text layer to remove; the scanned-document path is not part of this build.

What the held-out evaluation adds

Four worked rows show the mechanism, not the error rate. The evaluation case study measured both paths on a labeled synthetic test set with grouped splits (coding by preferred term, notes by template) and fitted calibration:

• Coding: right preferred term first for 78% of held-out verbatims on the real path against 54% for the stand-in; 38% auto-coded with no error at the default gate, against 25.5% with 7.1% of auto-codes wrong. Colloquial phrasing stayed unsolved.
• De-identification: 91 of 308 identifiers left readable on the real path and 108 on the stand-in, mostly 8-digit MRNs, 7-digit phone numbers, M/D/YYYY dates of birth and untitled names.
• Speed: about 0.4 s per verbatim and 0.9 to 1.6 s per note on a 4-vCPU CPU, one record at a time.

What the build does, and does not do

Does

• Records each run. One audit row per run with the files read, the column or columns used, the dictionary's source, row count and content hash, which path ran per tier (REAL or stand-in), every threshold in force, and on the real path a SHA-256 fingerprint of each loaded artifact. The QC suite checks that two runs on the same input give bit-identical output for the coding app (the real path too, when a models dir is given), the risk app and the monitoring app.
• Refuses to substitute silently. A tier shows REAL only when its artifact loads; an artifact whose runtime is missing, that fails to load, or whose feature contract the loaded tiers do not meet stops the run.
• Stays off the network on the real path. The apps set HF_HUB_OFFLINE=1, NEEDLE_TELEMETRY=0 and DO_NOT_TRACK=1 before any model loads and read only the models dir.
• Locks the public demo. The in-browser pages run in hosted mode: synthetic data only, every upload and path input disabled, and hosted_mode=true in the audit.

Does not

• Implement Part 11 controls, a review or sign-off step, or reviewer identity (see Read this first).
• Verify a MedDRA subscription. It records the dictionary it was given; MedDRA is a licensed product of the ICH MedDRA MSSO and none is included.
• Cover every identifier type. The stand-in's patterns find titled or surname-comma names, phone numbers, emails, one MRN format, two date-of-birth formats, ZIP codes, initials and "NN-year-old" ages; it has no pattern for addresses, SSNs or rare conditions. The output supports a de-identification review; it is not a HIPAA Safe Harbor or Expert Determination method and certifies nothing.
• Detect negation as a feature, derive clinical variables, process scanned documents, or make dosing or safety decisions. Reported study numbers belong to the sponsor's validated, double-programmed SAS or R programs and a medical coder's review, not to these apps.

Corrections to the retired decks

The two decks this page replaces said several things the build does not support. They are corrected here rather than kept. The first four rows were still in the decks when they were retired; the last four were in earlier versions of the decks and had already been replaced with measured values on 2026-10-22, before the decks were retired.

Check it yourself

• Stand-in values: open the Coding QC assist or the De-identification review assist in the browser and click Execute, or run either app locally against sample_data/. The span offsets, scores and sanitized text above are what they produce.
• Real-path values: build a models dir as in eval/README.md, set it as the Models dir and run again; ECRF_MODELS_DIR=<models dir> python tests/qc_test.py checks the run against eval/results/deck_b_worked.json.
• What was tested and fixed: QC_REPORT.md.