Trial‑Management Validation Harness

What the harness catches

A seeded-defect check (G10). Realistic programming defects from a list fixed in advance are planted one at a time in a throwaway copy of this repository. After each one the copy's outputs are regenerated, so G1's byte-for-byte check cannot fire; then the other gates and the module tests run, and the result is compared with an unmutated copy run the same way. Caught lists the layers that failed. Survived means no layer failed. Equivalent means no layer failed and no output or printed result differed (shown by diff, not assumed). The diff runs on the shipped data only, so Equivalent is not proof that a change has no effect: a change that matters only for data the sandbox does not contain is filed Equivalent. The list is hand-picked, so the counts describe this list only: they are not a detection rate and carry no interval. Survivors are reported as what the harness misses, not hidden.

Registry versions. Registry v1 (17 defects, D00 to D16) was frozen and committed before the first run, and all 17 outcomes matched its predictions (rows kept in teeth_2026-10-25_registry-v1.csv). A review probe then changed the convention grid's lower window bound from >= to >, so that total bilirubin on the same day as the aminotransferase draw no longer counted. No test failed and no output changed on the shipped data, so the runner filed it Equivalent: no planted case or study participant has its only qualifying TBL on that day. A same-day known-answer test was added, and registry v2 (18 defects) adds the change as D17, frozen before the second run. D17's prediction was written after the test that catches it, so its catch shows the gap is closed, not that the tests were well designed. Registry v2 also corrects one sentence of D04's rationale; no v1 prediction changed. Registry v3 changes only D04's prediction, from Survived to caught by G2, after G2 became a per-program check; it was frozen in its own commit before the third run.

What the 2026-10-25 run on registry v2 shows. G1 to G8 caught none of the seven numeric and counting defects (D05 to D11); only the separately written, same-author recomputation (G9) did. D04 survived: G2 then counted files, not programs, and passed while the log-scale pattern appeared in any file under r_tlf_pseudocode, documentation included, so it would have passed even if no R program used the log scale; nothing runs the TLF libraries on data. (G2 has since been changed to check, program by program, the R programs that compute geometric statistics or fit a BE or dose-proportionality model, found by content (28 today, with a committed floor of 28, so a program that leaves the set turns G2 red), and each one's SAS twin: in every branch, a geometric mean must be missing or exactly exp() of a log-scale mean (10^ in R or 10** in SAS for log10 logs), directly or through a name assigned that way, a geometric CV must come from a log-scale variance in the same way, and model responses must be log-derived, so a formula built from a string fails. A name assigned only a missing value is not a geometric name; in R a data frame that carries a log column is not itself log-scale, and a value passed through exp() is not log-scale either. Its unit tests catch D04, and the 2026-10-26 run on registry v3 caught D04 by G2: 15 caught, 0 survived, 1 equivalent, 2 not applicable, 18 of 18 as predicted, with every other outcome unchanged.) D01 was stopped by the generator's own consistency assertion before G3 could see it. The TRIALMON defects (D12 to D14, D17) were caught only by TRIALMON's tests, which run_harness.sh does not run. All 18 outcomes matched the predictions registered in advance, by this rule: same class, and every predicted catcher among the layers that caught it. More catchers than predicted still match: D12 and D13 were predicted to fail two TRIALMON test files and failed all four. The predictions came from the same author reading the gate code, so the match is not an independent test.

Registry: teeth/defects.csv (file, exact text replaced, replacement, predicted catcher and rationale per defect), frozen by its SHA-256 in teeth/defects.sha256 before each run (v1 6693bc70198f6a68c49ce9eee80f9dc053dec88ba07fa78f407703fd1cfa4e62; v2 e962ca98d30b2ac3b6c1db66d7271ea4f8beb2cb6b285606c2ea9f9c29d811d3); the runner refuses an edited registry. Runner: teeth/run_teeth.sh with teeth/apply_defect.py. Results: teeth/results/summary.json and the dated CSV beside it. G10 turns “stale” when the registry or any gate script changes after the run. Layers: G1 to G9 and the module tests (TRIALMON testthat files, the UAT logic test); “driver error” means run_lifecycle.R stopped with an error, and “generator error” that simulate_study() itself did.