How the coding and de-identification assists were evaluated

The failures, stated plainly

Results at a glance

Held-out test splits only; nothing was fitted on them. "Stand-in" is the default [SI] path that runs anywhere, including in the browser; "real" is the optional local model path.

What was measured, on what data

All evaluation data come from eval/make_data.py (seed 42; rerunning it reproduces eval/data/ byte for byte). Everything is invented: terms carry 99-prefixed placeholder codes, not MedDRA codes, and the groupings are not MedDRA's hierarchy.

How the shipped code was measured. eval/run_eval.py loads each app's own tier functions from its file (everything above its UI marker, through eval/_app.py), so the evaluation runs the code the apps ship rather than a copy. It runs the stand-in on the test split; builds the real Tier 2 and Tier 3 features on every split; fits the Laya temperatures on the calibration split; trains the XGBoost gate on the training split; writes those artifacts into a models dir; then reloads the app from that models dir and runs its real path on the test split. Those reloaded runs are the "real" numbers, and the script asserts that the reloaded app's calibrated probabilities match the ones the gate was trained on (to 0.002).

Scoring rules. Coding: top-1 and top-3 are measured on the 288 verbatims that have a correct code; the auto-code rate is over all 384; an auto-code is an error if its preferred term is wrong or the verbatim is negated. De-identification: a redaction counts as correct only on a strict match (same note, same start and end offset, same entity type); "left readable" counts gold targets with no overlapping redaction of any type, which forgives boundary and type slips and is the number that matters for privacy.

Grouped splits

A random row split would let the test set share terms and note templates with training, so a model could score well by memorizing them. The splits are grouped instead: coding rows by preferred term (a test term never appears in training or calibration) and notes by template (a test template is never seen in training).

Calibration

Laya answers each yes/no question with a probability: the softmax of its logits divided by a temperature. The checkpoint ships a temperature of 1.98; the script fits a new one on the calibration split with Laya's own fitting function and reports the test split before (shipped) and after (fitted). Expected calibration error (ECE) uses 10 equal-width bins; lower is better. A temperature changes how confident the probabilities are, not their order, so AUROC does not change.

Laya fits a temperature per answer type and option count only from 2,000 labeled records or more. The calibration splits here are smaller, so it fits one type-level temperature, and the two PHI questions share it. The indirect question's near-perfect AUROC reflects templated synthetic notes, an easy case, not a general result. The coding question's AUROC of 0.74 means Laya separates negated from plain verbatims only moderately; it is the re-ranker, combining that answer with the retrieval features, that kept every negated verbatim below the gate.

Coding: stand-in vs real

Test split: 384 verbatims (96 of each variant) coded against the 3,371-term dictionary. The real path retrieves the top 10 candidates with Needle3 and hnswlib, adds Laya's unambiguous probability, and re-ranks with a trained XGBoost classifier; the stand-in uses a hashed bag-of-words and a fixed weighted sum.

• The gain is mostly on misspellings (top-1 0.81 against 0.19), and the re-ranker auto-coded about half of them with no error. Exact matches were all auto-coded.
• The gate value did not matter on the real path here. The re-ranker's scores were bimodal (134 above 0.95, most of the rest below 0.5, none between 0.88 and 0.90). On the stand-in, the same 0.02 moved 6 negated verbatims across the gate.
• Case mattered. Needle3's embedding is case-sensitive ("insomnia" scored 0.948 against "Insomnia", below several unrelated terms). The first real run had this defect; the app now embeds lower-cased text on both sides, which is worth 0.125 top-1 on retrieval alone.
• Small dictionary. 3,371 terms is far sparser than a full release (about 86,800 lowest-level terms). Accuracy at release scale was not measured and should be expected to be lower.

Within-sibling discrimination

Many colloquial misses land in the right neighborhood and pick the wrong member: "calcium down on bloods" retrieves "Blood calcium increased" above "decreased". Top-1 accuracy mixes two failures, finding the neighborhood and choosing within it. This check isolates the second: the neighborhood is given, and the only question is which member the tool prefers. Everything below was registered before any scoring in eval/sibling_plan.md, which is locked by its SHA-256. Both scripts refuse to run when the plan or an input file differs from what was registered, and the evaluation script also refuses when the coding app's scoring code does.

• Sibling sets, fixed from preferred-term names: lab direction (44 analytes, each "increased", "decreased" and "abnormal") and body site (51 sites with four or six of "Pain in", "Swelling of", "Numbness of", "Tenderness of", "Weakness of" and "Cramp in").
• Items: the test-split verbatims whose correct term is in a set, 66 lab items (22 terms) and 144 site items (48 terms), three variants each. Negated verbatims have no correct code and are left out.
• The tool's choice: it scores every member of the set with its own shipped scorer and picks the highest. Within-set accuracy is the share picked right; the margin is the right member's score minus the best wrong member's.
• Chance, three ways, each relabeling the right answer within the set once per term: uniform (N1); by how often each member is the answer in training (N2); and, for colloquial items, uniform over the members whose words overlap the verbatim as much as the right one does (N3). The accuracy tail is exact; the margin's is Monte Carlo with 9,999 draws.
• A keyword rule ("low", "down", "below" for decreased, and so on), written into the plan before scoring, as a comparator.
• The gates decide auto-coding, not the p-values. Gate A: the one-sided 95% lower bound on within-set accuracy, at the effective size after clustering by term, must reach 0.90. Gate B: the tool must beat the keyword rule (one-sided 95% cluster-bootstrap lower bound above 0). A family that fails either gate, or could not reach 0.90 even with every item right, routes to human coding.

With every item right the bound is 0.05^1/m for m terms, so a 0.90 floor needs at least 29 terms and 0.95 needs 59. The locked plan's own table prints 0.940 for 48 terms and 0.951 for 59; the exact values round to 0.939 and 0.950. The scripts compute the bound themselves, so no decision used the printed values.

The last column is the stand-in's rank-1 accuracy over the whole dictionary on the same items, from the Stage 3 rows, for comparison.

• With the neighborhood given, the stand-in picks the right member most of the time (0.909 lab, 0.861 site, against a whole-dictionary top-1 of 0.636 and 0.549 on the same items). Most of its misses are in finding the neighborhood; colloquial verbatims carry the within-set misses.
• It beats chance but not lexical overlap. Both families beat N1 and N2 on accuracy and margin (p below 0.001). On colloquial items whose words overlap two or more members equally (11 lab, 22 site), accuracy is not distinguishable from chance under N3 (0.545, p = 0.12; 0.273, p = 0.08). The margin beats that null (p = 0.012 and 0.022) without separating the right member: on those items the mean margin is 0.007 on lab and -0.045 on site, where the right member is picked for only 6 of 22.
• It is not accurate enough to auto-code either family. Lab is too small to reach 0.90 with every item right, and its bound of 0.828 fails anyway; site's bound is 0.805.
• The tool was not shown to beat a keyword rule: the rule was ahead on lab (0.970 against 0.909); on site the tool was ahead (0.861 against 0.819), but its 0.042 lead was within bootstrap uncertainty (lower bound -0.014), so it failed Gate B. The rule's lexicon was written with the generator's templates visible, so this baseline is optimistic.
• Real path: not run in this pass (no models dir was available). The shipped re-ranker scores only candidates from the retrieved top 10, so the plan registers the Needle3 cosine as the real-path score. "calcium down on bloods" is picked correctly on the stand-in path; its Stage 3 failure is on the real path.

Every registered criterion was met. In the negative control, with the stand-in's scores shuffled among the members of each set 400 times, the accuracy test under N1 rejected 2.8% (lab) and 4.0% (site) of the time, against a registered ceiling of 7.2%. These criteria alone do not show that the gates work (row (e) still clears 93% with the design effect left out), so the test suite also checks the gates directly, outside the plan: the bound, the intraclass correlation and Gate B against hand-worked values, Gate B at the point where the tool and the rule are equally accurate (it passed in 4.5% of 1,000 such families), and the tails against enumeration.

Caveats. The dictionary is synthetic and not MedDRA, and its sibling families are regular by construction. The gold codes were generated, so they are independent of the tool by construction; real use needs gold codes adjudicated without the tool's suggestions, a licensed MedDRA release and a sibling rule registered before scoring. An "abnormal" colloquial verbatim such as "albumin off on labs" carries no direction, and its correct code is the "abnormal" term: code what is reported and query the site; nothing here infers a direction from linked lab values.

Prior art. Contrast sets (Gardner et al., Findings of EMNLP 2020) and CheckList (Ribeiro et al., ACL 2020) test models on minimally different inputs; hierarchical precision and recall (Kiritchenko et al. 2006), shortcut learning (Geirhos et al. 2020) and the conditional logit (McFadden 1974) are close relatives; the design effect (Kish 1965), the Clopper-Pearson bound (1934) and false discovery control (Benjamini and Hochberg 1995; Benjamini and Yekutieli 2001) are standard. What is new is modest: a base-rate-weighted within-set null, the keyword rule as a required comparator, and routing gates registered in advance with an attainability check. FDA's January 2025 guidance on AI to support regulatory decision-making is a draft, not for implementation, and the FDA-EMA good AI practice principles (January 2026) are non-binding; this check is not a validation under either.

Citation. Iben P. What do within-site comparisons of AlphaGenome Atlas predictions measure? An audit at ClinVar sites in 11 genes. Human Genomics, submitted. Preprint: doi:10.5281/zenodo.23543154. Code and data: doi:10.5281/zenodo.23537469. The full write-up is in EVALUATION.md.

De-identification: stand-in vs real

Test split: 220 notes from 10 templates never seen in training, holding 308 redaction targets. The real path proposes spans with Needle3's typed extraction plus the stand-in's patterns, and a trained XGBoost gate decides each one; the stand-in uses the patterns and a fixed rule. "Needle3 alone" uses Needle3's values directly as redactions, with no patterns and no gate, to show what the extractor contributes on its own.

P = precision, R = recall, strict match on offsets and entity type.

• The real path finds more names. Needle3 proposes names with no title in front ("Spoke with Linda Miller"), which the stand-in's patterns require, and the gate keeps most of them: name recall 0.33 to 0.74 at 0.94 precision.
• It lost some indirect attributes the stand-in kept (age recall 1.00 to 0.73, initials 1.00 to 0.89): the stand-in redacts any indirect attribute when two co-occur, and the trained gate learned a softer rule.
• The gate leans on span evidence, not on Laya. With direct_pii weak and biased low, the redactions that are right are right because of the span features.

Speed and index cost

One record at a time through each real tier, on the worked examples (25 timed calls after a warm-up; medians, ranges across four records):

Retrieval index (coding Tier 2): Needle3 vectors are 3072-dimensional. Over the 3,371-term dictionary the index held 40.4 MB, search took 0.54 ms (median) against 0.60 ms for exact brute force, and recall@10 against exact search was 0.994. At a full release's size, measured on 86,800 random unit vectors of the same width (index cost only, not retrieval quality), the index held about 1,040 MB and search took 5.8 ms. Embedding the verbatim (about 30 ms) costs more than the search.

Before the build: feasibility checks

Before any build work (Stage 0, 2026-10-21 and 2026-10-22), both models were downloaded, loaded offline on a CPU and checked against what the apps assumed. The full record is in FEASIBILITY.md. What it changed:

• The assumed embedding was wrong. The original design said "384-dim (SapBERT)"; Needle3's vectors are 3072-dimensional and it is not a biomedical encoder. Its similarity scores are compressed (unrelated terms scored 0.93 against 0.96 for the right one), so no fixed cosine cutoff carries over.
• Laya is over-confident as shipped (its authors say so) and near chance zero-shot on its authors' decision benchmark. One measured example went the wrong way: "Does the patient report a headache?" on "Patient denies headache." gave 0.655. That is why calibration and a labeled negation set were required before any Laya output drove a decision.
• Five wiring defects in the apps would have stopped or mislabeled the real path: the model modules were never bound to names; the Laya call used route(), which returns a routing decision, not answers; the banners claimed REAL when only a library was importable; the guarded artifact names (needle3.onnx, laya.onnx) do not exist in either model repository; and the Laya probe reported REAL with torch absent. All five were fixed, and each fix is pinned by an always-run check in tests/qc_test.py (RW1 to RW16, stub packages, no models needed).
• Network and telemetry. Needle3's Python package sends usage telemetry unless it is switched off, and its loader asks the Hugging Face Hub for a config file on every load, even with the weights cached. The apps set NEEDLE_TELEMETRY=0, DO_NOT_TRACK=1 and HF_HUB_OFFLINE=1 before any model loads and read only the models dir. An unrelated PyPI package also installs the import name needle; the apps reject it.

Where the numbers come from

Measured on 2026-10-22 on one cloud container: Intel Xeon @ 2.80GHz, 4 vCPU, no GPU, 15 GB RAM, Linux 6.18, Python 3.12.3, with cactus-needle 3.2.8 (engine 3.3.3), Laya 0.4.13 (English checkpoint convaiinnovations/laya at commit 55cf4c4e), hnswlib 0.8.0 (M 32, efConstruction 200, efSearch 64), xgboost-cpu 3.4.1, torch 2.14.1+cpu and transformers 5.20.2. The stand-in numbers come from the same runs. Everything is seeded and CPU inference is deterministic, so a rerun on the same versions reproduces the committed files. To rerun, follow eval/README.md: install the pinned runtimes, build a models dir with python eval/fetch_models.py --out models, then:

python eval/make_data.py                        # regenerate eval/data/ (byte-identical)
python eval/bench_index.py --models models      # index cost
python eval/run_eval.py   --models models       # calibrate, train, evaluate
python eval/deck_b_worked.py --models models    # worked examples, per-record latency
ECRF_MODELS_DIR=models python tests/qc_test.py  # adds the RM real-path checks
python eval/sibling_eval.py --models models     # within-sibling check (without --models: stand-in only)

The long-form report, with measurement notes, is EVALUATION.md.

What this does not show

• Real data. Accuracy on a licensed MedDRA release or on real site notes is unknown and should be expected to be lower.
• Retrieval quality at release scale. The 86,800-term figures are index cost on random vectors.
• The within-sibling check on the real path. It ran on the stand-in path only; it needs a models dir.
• Participant risk screening and Central statistical monitoring. Neither has a trained-model path or a labeled evaluation, so no accuracy, sensitivity or false-positive rate was measured for either. They run labeled rule-based stand-ins (plus optional scikit-learn and POT detectors in the monitoring app), checked only against signals planted in the synthetic data, by the QC suite in tests/qc_test.py. Risk screening: of the 52 participants in the bundled cut, the 5 test-case participants the suite checks get the category or reason they were built for (fatal SUBJ-9015 and multi-system SUBJ-9001 CRITICAL, severe-AE SUBJ-9002 HIGH, an ALT breach cited for SUBJ-9013 and a QTc breach for SUBJ-9014; checks R10 to R16). Monitoring, on the 150-participant longitudinal demo as the in-browser page runs it: all 3 injected numeric outliers are flagged, plus 1 participant with no injected outlier (SUBJ-A150, MODERATE at z = 3.0, a side effect of the injected LAB-C AST shift; check A8c), and exactly the 3 injected unusual notes (A14). Known answers like these show the checks fire on what was planted; they do not estimate how often a real signal is missed or a clean participant is flagged.
• Validation. These are evaluated prototypes, not validated systems. Reported study numbers belong to the sponsor's validated, double-programmed SAS or R programs; nothing here replaces that or the medical coder's and privacy reviewer's judgment.