eCRF Pipeline Apps

Try a hosted demo

Each app runs as its own Streamlit Community Cloud deployment in hosted mode: every data input is locked to the bundled synthetic cut, outputs go to a per-session temporary folder, a banner says so on every page, and the audit records hosted_mode=true. For real data, run the apps locally (see the README).

Coding app (meddra_app.py): MedDRA-style auto-coding of adverse-event verbatims
PHI app (phi_app.py): PII/PHI flagging and redaction
Risk app (risk_app.py): participant safety and discontinuation risk scoring
Anomaly app (anomaly_app.py): numeric outliers, distributional drift and text outliers

Each opens on the bundled synthetic sample data; click Execute to run it. An app that has been idle may take about 30 seconds to wake up.

The four apps

The coding, PHI and risk apps read the same synthetic cut folder without interfering; the anomaly app reads its own longitudinal demo, because drift needs a time axis. All four accept CSV, and SAS .sas7bdat when pyreadstat is installed.

Auto-coding

meddra_app.py

Reads: The one verbatim column you select (AETERM or a synonym), plus a dictionary as a flattened CSV or an .asc release.
Writes: ae_coded.csv (top three candidates per verbatim, with the LLT to SOC path) and audit_log.csv.

PII/PHI redaction

phi_app.py

Reads: Every non-numeric column, found by content rather than by name. No dictionary.
Writes: phi_flags.csv (one row per scrubbed span), sanitized/<domain>.csv and audit_log.csv.

Risk scoring

risk_app.py

Reads: Safety domains (ae, lb, eg, vs, pe), classified by content and scored against built-in [SI] criteria. No SAP file.
Writes: risk_scores.csv (safety and discontinuation scores with contributing factors) and audit_log.csv. Decision support, not a clinical determination.

Anomaly and drift

anomaly_app.py

Reads: Longitudinal files aligned by USUBJID: numeric columns, plus optional visit, lab/site and free-text columns.
Writes: anomalies.csv, drift.csv, lab_drift.csv, lab_compare.csv, text_anomalies.csv and audit_log.csv.

The four-tier chain

Every app runs the same discipline: four tiers in sequence, each handing named, readable columns to the next, so a decision is a row of numbers a reviewer can inspect and its threshold is a parameter a reviewer can move.

Tier 1, intake. Aligns each row to its subject, copies every value through unchanged (missing stays missing) and locates the columns to read by content.
Tier 2, retrieval and extraction. Proposes and measures: candidate dictionary terms, typed identifier spans, or per-subject signals. It decides nothing.
Tier 3, context and calibration. Adds calibrated probabilities, an ambiguity score, aggregate risk scores or drift distances.
Tier 4, decision. Applies an explicit rule to the earlier tiers' columns only: the auto-commit gate (default 0.88), the redaction gate (default 0.85), or the risk and anomaly categories.

Stand-in `[SI]` versus real

No trained weights ship with this repository, so the model tiers run as labeled deterministic stand-ins, tagged [SI] in the banner and in every audit row, with the same output columns the real components would produce. Run locally with the open-source runtimes (Cactus Needle3, hnswlib, Laya, XGBoost) and a models dir, and the coding and PHI apps switch to their real path tier by tier, air-gapped; a tier shows REAL only when its artifact loads, and an artifact that cannot be used stops the run rather than being replaced by a stand-in. What the real path does on synthetic data is measured in the evaluation: better than the stand-in on misspellings and untitled names, no better on colloquial verbatims, and about 0.4 s per verbatim on a CPU. Optional packages also turn on real paths in the anomaly app: scikit-learn adds an Isolation Forest, and POT with scipy runs Sinkhorn optimal-transport drift.

Screenshots

Local runs on the bundled synthetic data, opened with ?demo=1 and captured after Execute. Select an image to open it at full size.

Coding app results table: each adverse-event verbatim with its top three candidate terms, ranks and 99-prefixed placeholder codes, followed by the success message and download buttons. — Coding app: top three candidates per verbatim, with placeholder codes; the ranking comes from the `[SI]` lexical stand-in.

PHI app results: a table of flagged spans such as invented phone numbers, names, a ZIP code and initials, each with its entity type, offsets, redaction probability and action, above the sanitized domain rows. — PHI app: one row per scrubbed span; every name and number shown is invented.

Risk app results: per-subject safety and discontinuation scores and categories with contributing factors, the fatal subject pinned to the top, and a category summary. — Risk app: per-subject safety and discontinuation scores, sorted highest risk first, with the reasons for each.

Anomaly app results: flagged per-subject numeric outliers with their extreme columns, a cohort-wide drift table per column, and the start of the per-lab drift table. — Anomaly app: flagged numeric outliers, cohort-wide drift per column and per-lab drift on the longitudinal demo.

Presentations

Deck A: The Four-Tier Pipeline. The conceptual deck: why the work is split into four tiers, and the governance and limits.
Deck B: Implementation Walkthrough. Two worked traces, coding and PHI, on the synthetic cut.

Use the arrow keys or the buttons at the bottom of each deck to move between slides. Links inside the decks to their Markdown versions and to other repository files work only from the decks in presentations/ in the GitHub repository, not from this site.

About this page

This is a single static file, docs/index.html, with no external scripts, stylesheets or fonts. GitHub Pages serves it from the main branch's /docs folder.