PRISM-EHR

A synthetic electronic health record corpus and end-to-end validation pipeline, benchmarked across four clinical cohorts.

PRISM-EHR is two things in one repository:

1. A released synthetic-EHR dataset. A corpus of 640 paired synthetic patient tables (16 benchmark entries, including a Train passthrough, × 10 frozen 70/30 splits × 4 cohorts), distributed under CC BY 4.0.
2. An end-to-end validation and evaluation pipeline that produced the corpus and that downstream researchers can re-use on any binarized tabular dataset conforming to the documented schema. The pipeline covers data ingest, concept extraction and clinical-threshold binarization, frozen split construction, model training and generation, fidelity and disclosure-risk metric computation with 95% percentile-bootstrap intervals, paired non-parametric significance testing, and use-case-aware rank composition under five deployment scenarios.

This repository accompanies the PRISM-EHR Data Descriptor (Nature Digital Medicine). It is the canonical source code referenced by the manuscript's Code Availability statement and contains the full data, model implementations, evaluation metrics, and statistical analyses needed to reproduce every figure and table in the paper.

Cohorts

PRISM_DATASET	Source	Population	Final $n$
nhanes_2015 (default)	CDC NHANES 2015–2016	US cross-sectional health survey	4,809
nhanes_2017	CDC NHANES 2017–2018	US cross-sectional health survey, second cycle	4,479
eicu	PhysioNet eICU-CRD v2.0	US multi-centre critical care	60,494
zigong	PhysioNet Zigong Fourth People's Hospital v1.3	China, inpatient heart failure	2,008

PRISM_DATASET selects the cohort for every stage of the pipeline; see docs/gpu_models_runbook.md for the full per-cohort command reference. Examples throughout this README default to nhanes_2015 (the unset default) — substitute PRISM_DATASET=<cohort> for the others.

Audience: reviewers verifying the released artifacts, researchers re-using the synthetic corpus as a benchmark substrate for new generators, and practitioners applying the pipeline to other public-use clinical datasets.

---

Table of contents

1. Overview
2. Repository layout
3. Software environment
4. Hardware notes
5. Reproduction pipeline
6. Output artifacts
7. Determinism and seeds
8. Metrics and statistical procedure
9. Extending the benchmark with a new model
10. Data provenance and licensing
11. Contact

---

1. Overview

The PRISM-EHR pipeline validates every released synthetic dataset against the held-out test half of its split along three axes:

• Fidelity — Dimension-Wise Distribution (DWD), Column-Wise Correlation (CWC), Latent Cluster Analysis (LCA)
• Privacy — Attribute Inference Risk (AIR), Membership Inference Risk (MIR), Meaningful Identity Disclosure Risk (MIDR), Nearest-Neighbor Adversarial Accuracy (NNAA)
• Aggregate ranking — Rank Correlation Analysis (RCA) under five weighting schemes (Equal, Education, Medical AI Development, System Development, Clinical Research)

Every model is refit on each of 10 frozen 70/30 train/test splits of each of the four cohorts, producing paired per-split observations for hypothesis testing. Fifteen generators were trained, sampled, and evaluated per cohort; a Train passthrough is included to calibrate metric behavior under no generator-induced distortion (16 entries total). A separate cross-database analysis (scripts/experiments/cross_database_analysis.py) pools ranks across all four cohorts to test whether the generator ordering is cohort-specific.

Models compared

16 benchmark entries across five categories (real, the Train passthrough, plus 15 fitted generators):

Category	Identifier	Description
Reference	real	Train passthrough (training half scored against the test half)
Reference	baseline	Independent Marginal Sampler (IMB)
Probabilistic	bn	Chow-Liu Bayesian Network
Probabilistic	mob	Mixture of Bernoullis (MoB)
Probabilistic	gcopula	Gaussian Copula (GCop)
Probabilistic	logreg_chain	Logistic Regression Chain (LRC)
Neighbour	dt_knn	Decision Tree + k-NN sampler
Neighbour	synthpop	Sequential CART (SCART)
Deep	vae	Variational Autoencoder
Deep	medgan	MedGAN
Deep	corgan	CorGAN
Deep	ddpm	Denoising Diffusion Probabilistic Model
Language	distilgpt2	DistilGPT-2 (82M params) via GReaT
Language	gpt2	GPT-2 small (124M) via GReaT
Language	gpt2_medium	GPT-2 medium (355M) via GReaT
Language	gpt2_large	GPT-2 large (774M) via GReaT

---

2. Repository layout

PRISM-EHR/
├── README.md                 ← this file (reproducibility guide)
├── requirements.txt          ← pinned-floor Python dependencies
│
├── slurm/
│   └── site_config.example.sh  ← cluster configuration template (copy + edit)
│
├── data/
│   ├── <cohort>/              ← nhanes_2015, nhanes_2017, eicu, zigong
│   │   ├── raw/               ← source .XPT/.csv/.htm downloads (local only, not released)
│   │   ├── preprocessed/
│   │   │   ├── full/          ← master_concepts.csv, binary_matrix.{npy,csv} (pre-split source)
│   │   │   └── splits/split_XX/  ← 10 frozen train/test splits + binary siblings
│   │   ├── synthetic/
│   │   │   └── split/<model>/split_XX/  ← synthetic outputs (15 models × 10 splits)
│   │   └── evaluation/split/  ← per-split + aggregated metric JSONs and plots
│   └── _cross_database/      ← pooled cross-cohort Friedman/rank analysis
│
├── scripts/
│   ├── scrape_data/          ← per-cohort data acquisition (nhanes/, eicu/, zigong/)
│   ├── preprocess_data/      ← merge → recode/bin → one-hot encode
│   └── experiments/          ← split builders, classical runner, neural runner,
│                                table builders, significance tests, cross-database analysis
│
├── models/                   ← model dirs grouped by category (see evaluation/config.py:MODELS)
│   ├── <category>/<classical>/       ← model.py — sampler class invoked by run_classical_split.py
│   ├── <category>/<deep-generative>/ ← model.py + train.py + generate.py + slurm/
│   └── language/slm/                 ← train.py + generate.py + slurm/ (GReaT-based SLMs)
│
├── evaluation/
│   ├── config.py             ← single source of truth for paths, seeds, params
│   ├── run.py                ← python -m evaluation.run
│   ├── plot.py               ← python -m evaluation.plot
│   ├── utils/
│   │   ├── data_loader.py    ← per-split one-hot loaders
│   │   └── uncertainty.py    ← bootstrap CIs
│   └── metrics/<group>/<metric>/{compute.py, plot.py}
│       ├── fidelity/         ← dwd, cwc, lca
│       ├── privacy/          ← air, mir, midr, nnaa
│       └── recommendation/   ← rca
│
└── docs/
    └── new_models_runbook.md ← step-by-step guide for adding a new model

---

3. Software environment

• Python: 3.10 or newer (tested on 3.10–3.13).
• Dependencies: see requirements.txt. Floor versions are listed; pinning the exact versions used in the paper is recommended for byte-identical reproduction.
• Recommended setup (CPU-only metrics + classical/statistical models):

git clone https://github.iu.edu/skyangel/PRISM-EHR.git
cd PRISM-EHR
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

For the deep-generative and SLM training/generation steps you additionally need a CUDA-capable GPU and a working torch install matching your driver version.

---

4. Hardware notes

Stage	Hardware used in the paper	Approximate wall-clock
Preprocessing (scripts/preprocess_data/)	any modern laptop	< 1 minute
Split building (make_splits.py, make_binary_splits.py)	any modern laptop	< 1 minute
Classical/statistical generation (10 splits × 7 models)	8-core CPU, 16 GB RAM	≈ 15 minutes total
VAE training (10 splits)	1× NVIDIA L40S 48GB	≈ 30 min per split
MedGAN training (10 splits)	1× NVIDIA L40S 48GB	≈ 45 min per split
CORGAN training (10 splits)	1× NVIDIA L40S 48GB	≈ 45 min per split
DDPM training (10 splits)	1× NVIDIA L40S 48GB	≈ 60 min per split
SLM training (10 splits × 4 sizes, 200 epochs each)	1× NVIDIA H100 80GB or L40S 48GB per array task	4–12 hours per (model, split) — submitted as a SLURM array under models/language/slm/slurm/
SLM generation (4809 samples × 10 splits × 4 sizes)	same as training	≈ 10–60 minutes per (model, split)
Evaluation (evaluation.run + evaluation.plot)	8-core CPU, 32 GB RAM	≈ 15 minutes for all metrics × all models × 10 splits
Significance tests (scripts/experiments/significance_testing.py)	same	< 1 minute

The deep-generative and SLM stages are the only steps that require a GPU. Without GPU resources, reviewers can still reproduce all classical and statistical model results and re-run the full evaluation pipeline against the released synthetic data in data/<cohort>/synthetic/split/.

---

5. Reproduction pipeline

The pipeline is deterministic given the seeds in evaluation/config.py and scripts/experiments/make_splits.py. Run from the repository root. Raw source data is never committed to this repository (NHANES is public-use but not re-hosted here for size reasons; eICU and Zigong are PhysioNet credentialed-access datasets whose data-use agreements prohibit redistribution outside PhysioNet). Reviewers reproducing from scratch must acquire each cohort's raw data themselves before preprocessing.

5.1. Acquire the raw cohort data

# NHANES (public, no credentialing required)
python3 scripts/scrape_data/nhanes/1_scrape_files.py     # downloads .XPT + .htm
python3 scripts/scrape_data/nhanes/3_convert_to_csv.py   # XPT → CSV
PRISM_DATASET=nhanes_2017 python3 scripts/scrape_data/nhanes/1_scrape_files.py
PRISM_DATASET=nhanes_2017 python3 scripts/scrape_data/nhanes/3_convert_to_csv.py

# eICU-CRD and Zigong (PhysioNet credentialed access — requires your own approved
# PhysioNet account and completed data-use agreement before these will authenticate)
PRISM_DATASET=eicu python3 scripts/scrape_data/eicu/acquire.py
PRISM_DATASET=zigong python3 scripts/scrape_data/zigong/acquire.py

The variable index used to confirm NHANES column codes is scraped by scripts/scrape_data/nhanes/2_scrape_codes.py. See scripts/scrape_data/README.md for per-cohort acquisition detail.

5.2. Preprocess the cohort

# NHANES recode; see scripts/preprocess_data/README.md for eICU/Zigong equivalents.
python3 scripts/preprocess_data/nhanes/1_merge.py       # merge sources, filter to age ≥ 20
python3 scripts/preprocess_data/nhanes/2_recode_bin.py  # recode + bin → master_concepts.csv
python3 scripts/preprocess_data/common/binary_matrix.py # one-hot → binary_matrix.{npy,csv} (shared)

Prefix any command in this section with PRISM_DATASET=<cohort> to target nhanes_2017, eicu, or zigong instead of the default nhanes_2015.

5.3. Build the 10 frozen splits

python -m scripts.experiments.make_splits           # writes data/nhanes_2015/preprocessed/splits/split_{00..09}/
python -m scripts.experiments.make_binary_splits    # adds train_binary.csv siblings for classical models

make_splits.py uses random_state = 42 + split_idx for each split; the output directories include indices.json so that any external reviewer can verify row membership.

5.4. Train and generate synthetic data

As in §5.1–5.2, prefix any command below with PRISM_DATASET=<cohort> to target a cohort other than the default nhanes_2015; SLURM submissions pass it through --export. See docs/gpu_models_runbook.md for the full per-cohort reference.

Classical and statistical models (CPU, all 10 splits):

python -m scripts.experiments.run_classical_split
# Runs baseline, bn, dt_knn, synthpop, mob, logreg_chain, gcopula × 10 splits.

Deep generative models on a single local GPU (VAE, MedGAN, CORGAN, DDPM):

python -m scripts.experiments.run_neural_split
# Trains and generates for vae, medgan, corgan, ddpm × 10 splits sequentially.
# Options: --models vae,ddpm  --splits 0,1,2  --epochs 50  --skip-train  --overwrite

Deep generative models and SLMs on HPC (SLURM array jobs):

First, configure your cluster site (one-time setup):

cp slurm/site_config.example.sh slurm/site_config.sh
# Edit slurm/site_config.sh: set SLURM_ACCOUNT, SLURM_PARTITION, CONDA_INIT, CONDA_ENV

Then submit train + generate arrays for each model (example: VAE):

bash models/deep/vae/slurm/submit/submit_train_split.sh vae
bash models/deep/vae/slurm/submit/submit_generate_split.sh vae

Repeat for ddpm, corgan, medgan (same pattern). For SLMs:

bash models/language/slm/slurm/submit/submit_train_split.sh distilgpt2
bash models/language/slm/slurm/submit/submit_generate_split.sh distilgpt2
# Repeat for gpt2, gpt2_medium, gpt2_large

Each submit script launches a --array=0-9 SLURM job covering all 10 splits.

5.5. Evaluate

python -m evaluation.run     # per-split metrics + cross-split aggregation + RCA
python -m evaluation.plot    # all paper figures

5.6. Reproduce the paper tables

python -m scripts.experiments.build_latex_table        # Table: per-cohort results
python -m scripts.experiments.significance_testing     # Tables: Friedman, Wilcoxon, Page's L
python -m scripts.experiments.cross_database_analysis   # Tables/figures: pooled cross-cohort ranking

The cross-database step reads every cohort's aggregated metrics under data/<cohort>/evaluation/split/, so run §5.1–5.5 for all four cohorts first.

---

6. Output artifacts

After a complete run, the following files exist per cohort (all paths relative to the repo root, <cohort> is one of nhanes_2015, nhanes_2017, eicu, zigong):

Path	Contents
data/<cohort>/preprocessed/full/binary_matrix.{npy,csv}	One-hot pre-split cohort matrix (input to make_splits.py) — local only, not released
data/<cohort>/preprocessed/splits/split_XX/	train.csv, test.csv, indices.json, plus *_binary.* siblings — local only, not released
data/<cohort>/synthetic/split/<model>/split_XX/synthetic.csv	Per-model synthetic data for each split — released
data/<cohort>/evaluation/split/per_split/split_XX/<metric>_results.json	Per-split raw metric values — released
data/<cohort>/evaluation/split/<metric>_aggregated.json	Mean/std/95% bootstrap CI across splits — released
data/<cohort>/evaluation/split/rca_results.json	Aggregate RCA rankings per weight scheme — released
data/<cohort>/evaluation/split/plots/*.png	All figures (DWD, CWC, LCA, AIR, MIR, MIDR, NNAA, RCA, heatmaps) — released
data/<cohort>/evaluation/split/results_table.tex	Headline LaTeX results table — released
data/<cohort>/evaluation/split/significance_table.tex	Hypothesis-test LaTeX tables — released
data/<cohort>/evaluation/split/significance_results.json	Machine-readable significance output — released

data/<cohort>/raw/ and data/<cohort>/preprocessed/ are never committed to this repository or the released dataset: raw source data is regenerated locally from §5.1–5.2, and only synthetic/ and evaluation/ are tracked and released. data/_cross_database/ holds the pooled cross-cohort Friedman/rank outputs from §5.6.

---

7. Determinism and seeds

All randomness is centralised. The seeds below appear in evaluation/config.py (or are derived from split_idx) and are sufficient to reproduce every paper result:

Component	Seed
Train/test split builder	42 + split_idx
Classical/statistical sampler RNGs (per split)	42 + split_idx
Deep generative training (per split)	42 + split_idx (overridable via --seed)
SLM training (per split)	42 + split_idx (overridable via --seed)
Bootstrap resamples (CI + significance)	42
N_SPLITS	10 (every cohort)
N_SAMPLES per synthetic dataset	equal to that cohort's n — 4,809 (nhanes_2015), 4,479 (nhanes_2017), 60,494 (eicu), 2,008 (zigong)

---

8. Metrics and statistical procedure

• Fidelity (lower is better): DWD, CWC, LCA per Yan et al., Nat. Commun. 13:7609 (2022).
• Privacy (lower is better):
  • AIR — k-NN attribute inference attack on training records (k=1, top-32 known attributes).
  • MIR — Euclidean distance to nearest synthetic neighbor at θ ∈ {1.0, √2, 2.0, 5.0}; main θ = 1.0.
  • MIDR — Meaningful Identity Disclosure Risk; extends the Yan et al. framework to quasi-identifier combinations to measure re-identification risk from sensitive attribute linkage.
  • NNAA — Nearest-Neighbor Adversarial Accuracy; measures the balance between train-set (AA_ES) and test-set (AA_TS) attack accuracy — a value near 0.5 indicates privacy-preserving generation.
• RCA: per-metric ranks across models, weighted by five deployment-scenario weight schemes; ties broken by average rank.
• Uncertainty: 95% percentile bootstrap CIs over the 10 per-split values, 10,000 resamples, seed 42 — applied uniformly to plots and tables.
• Significance (scripts/experiments/significance_testing.py): Friedman omnibus per metric (gates the pairwise families), paired Wilcoxon signed-rank with Holm correction within each metric, Page's L trend test on the four SLM sizes. Pre-registered alternatives are documented in the script's module docstring.

---

9. Extending the benchmark with a new model

See docs/new_models_runbook.md for a detailed walkthrough. The summary:

1. Create models/<model_key>/ with model.py exposing a sampler class with fit(X_bin, feature_names=...) and sample(n) methods. For CPU samplers, dt_knn/ and gcopula/ are minimal references. For GPU-trained models, models/deep/vae/ is the reference (includes train.py + generate.py + slurm/ templates).
2. Read inputs from data/nhanes_2015/preprocessed/splits/split_XX/{train.csv, train_binary.csv}.
3. Write outputs to data/nhanes_2015/synthetic/split/<model_key>/split_XX/synthetic.csv. Output columns must match master_concepts.csv exactly.
4. Register the model in evaluation/config.py by adding "<model_key>": "<display_name>" to MODELS.
5. For CPU models: add the model key and runner to scripts/experiments/run_classical_split.py. For GPU models: add the model to scripts/experiments/run_neural_split.py and create sbatch templates under models/<model_key>/slurm/ (copy models/deep/vae/slurm/ as a starting point).
6. Re-run python -m evaluation.run (or --model <model_key> for a focused pass) and python -m evaluation.plot.

---

10. Data provenance and licensing

• NHANES 2015–2016 and 2017–2018: Centers for Disease Control and Prevention. Public domain (US federal data). Adults aged ≥ 20, complete-case analysis on 21 benchmark concepts (8 categorical + 13 binary diagnoses). Cohort sizes after filtering: 4,809 (2015–2016), 4,479 (2017–2018). Variables retained are listed in scripts/preprocess_data/nhanes/1_merge.py:SOURCE_FILES.
• eICU Collaborative Research Database v2.0: PhysioNet, credentialed access. Requires an approved PhysioNet account and a completed data-use agreement to acquire the raw source; that agreement prohibits redistributing the raw or row-level de-identified data outside PhysioNet's own system. One ICU stay per patient, 22 concepts, complete-case cohort size 60,494.
• Zigong Fourth People's Hospital heart-failure cohort v1.3: PhysioNet, credentialed access, same redistribution restriction as eICU. 22 concepts, cohort size 2,008 (every hospitalisation survives complete-case handling).
• What this repository and its released dataset contain: only the pipeline code and the derived synthetic data plus evaluation outputs for all four cohorts. No raw or row-level real patient data for any cohort is committed to this repository or included in the released Zenodo dataset — reviewers reproducing from raw source must acquire eICU and Zigong themselves under their own PhysioNet credentials (§5.1).
• PHI/identifiers: all four source cohorts are de-identified at source. The synthetic-data privacy attacks in this benchmark target the de-identified training rows, not real individuals; results bound risk against this de-identified surface only.

---

11. Contact

Sky Angeles — Corresponding author
skyangel@iu.edu