Synthetic Data ScoreCard: A Data Quality Evaluation Toolbox for Synthetic Medical Datasets

Overview

Synthetic Data ScoreCard is an open-source Python toolbox that implements the synthetic data evaluation approach for assessing the quality of Synthetic Medical Data based on 7C's approach as described in [Zamzmi et al. (2025)].

Synthetic medical data offers potential advantages for AI development in healthcare — particularly for addressing data scarcity, privacy constraints, and underrepresentation of rare diseases and populations. However, the quality of synthetic data is critical: poor-quality synthetic data may introduce biases, violate clinical constraints, and potentially affect the performance of AI models trained or evaluated on such data.

This toolbox provides a quantitative and reproducible approach to evaluating Synthetic Medical Data across seven relevant criteria, and supports the generation of a structured Scorecard to accompany synthetic datasets.

For more information, and technical questions please contact: Seyed.Kahaki@fda.hhs.gov, Alexander.Webber@fda.hhs.gov, or Tahsin.Rahman@fda.hhs.gov

---

The Synthetic Data Evaluation Approach

The SMD ScoreCard evaluates synthetic medical data across the following seven dimensions:

1. Congruence — Measures the degree to which the distribution of synthetic data aligns with the distribution of real patient data (e.g., using Earth Mover's Distance, Cosine Similarity, Jensen-Shannon Divergence).

2. Coverage — Evaluates the extent to which SMD captures the coverage, and range inherent in patient data (e.g., using Convex Hull Volume, Variance, Distance to Centroid).

3. Completeness — Evaluates whether the generated data contains all necessary details relevant to the intended task (e.g., using Proportion of Required Fields, Missing Data Percentage compared to a reference dictionary).

4. Consistency — Assesses the stability of SMD Completeness across demographic subgroups, disease classes, or over time (e.g., using Variance, ANOVA, Maximum-Minimum Difference).

5. Constraint — Assesses adherence to known anatomical, biological, geometric, or user-defined constraints (e.g., using Constraint Violation Rate to measure how often synthetic features fall outside of ranges defined by corresponding real features).

6. Comprehension — Checklist to evaluate the transparency and clarity of the data generation process as well as the quality of the accompanying documentation.

7. [Compliance] — Synthetic dataset providers are encouraged to ensure compliance of their datasets with applicable regulations and institutional policies. Note: Compliance is not evaluated quantitatively by this toolbox. Dataset providers are responsible for ensuring compliance with applicable regulations.

ScoreCard — Agglomeration of all results into one notebook dashboard. Run after running previous notebooks.

---

Synthetic Medical Data Scorecard Structure

In addition to quantitative evaluation, this toolbox supports generation of a structured SMD Scorecard report containing the following sections:

1. General Information — Dataset name, modality, size, labels, licensing, and point of contact.
2. Data Quality Evaluation — Quantitative results for each of the seven criteria.
3. Dataset Usage — Repository links, preprocessing requirements, and user documentation.
4. Training & Validation Process — Description of the synthetic data generation pipeline.
5. Reference Dataset Information — Key details of the patient dataset used for comparison.

---

Getting Started

Installation & Startup

• Clone this repository and navigate to the project directory

• Ensure you are using Python 3.12

• Install requirements: pip install -r requirements.txt

• Use the IPython notebooks in notebooks to visualize and analyze the pre-extracted features

• Run the Flask application: python app.py

• Navigate to: http://localhost:5050 in a browser to access the application

Note for the Flask application: The image files specified in data/real_patch_appearance.csv and data/static_patch_appearance.csv need to be made available in the appropriate paths outlined in the csv for the Preset Dataset option to be used.

---

Contact

For any inquiries or suggestions, please contact Seyed Kahaki, Elim Thompson, or Aldo Badano either via this GitHub repo or via email (seyed.kahaki@fda.hhs.gov, yeelamelim.thompson@fda.hhs.gov or aldo.badano@fda.hhs.gov).

---