Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges

Abstract

This paper presents a comprehensive systematic review of generative models (GANs, VAEs, DMs, and LLMs) used to synthesize various medical data types, including imaging (dermoscopic, mammographic, ultrasound, CT, MRI, and X-ray), text, time-series, and tabular data (EHR). Unlike previous narrowly focused reviews, our study encompasses a broad array of medical data modalities and explores various generative models. Our search strategy queries databases such as Scopus, PubMed, and ArXiv, focusing on recent works from January 2021 to November 2023, excluding reviews and perspectives. This period emphasizes recent advancements beyond GANs, which have been extensively covered previously. The survey reveals insights from three key aspects: (1) Synthesis applications and purpose of synthesis, (2) generation techniques, and (3) evaluation methods. It highlights clinically valid synthesis applications, demonstrating the potential of synthetic data to tackle diverse clinical requirements. While conditional models incorporating class labels, segmentation masks and image translations are prevalent, there is a gap in utilizing prior clinical knowledge and patient-specific context, suggesting a need for more personalized synthesis approaches and emphasizing the importance of tailoring generative approaches to the unique characteristics of medical data. Additionally, there is a significant gap in using synthetic data beyond augmentation, such as for validation and evaluation of downstream medical AI models. The survey uncovers that the lack of standardized evaluation methodologies tailored to medical images is a barrier to clinical application, underscoring the need for in-depth evaluation approaches, benchmarking, and comparative studies to promote openness and collaboration.

References in corpus (78)

• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Conditional Generative Adversarial Nets
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Denoising Diffusion Probabilistic Models
• LLaMA: Open and Efficient Foundation Language Models
• Neural Discrete Representation Learning
• Unsupervised Data Augmentation for Consistency Training
• A Survey of Large Language Models
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Scaling Instruction-Finetuned Language Models
• Classifier-Free Diffusion Guidance
• Random Erasing Data Augmentation
• Publicly Available Clinical BERT Embeddings
• Efficiently Modeling Long Sequences with Structured State Spaces
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Metrics reloaded: Recommendations for image analysis validation
• C-RNN-GAN: Continuous recurrent neural networks with adversarial training
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Diffusion Probabilistic Models beat GANs on Medical Images
• Differentially Private Synthetic Medical Data Generation using Convolutional GANs
• Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN
• CBCT-Based Synthetic CT Image Generation Using Conditional Denoising Diffusion Probabilistic Model
• MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
• Natural Language Generation for Electronic Health Records
• Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
• SADM: Sequence-Aware Diffusion Model for Longitudinal Medical Image Generation
• RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
• Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models
• Multi-Label Clinical Time-Series Generation via Conditional GAN
• Synthesize High-dimensional Longitudinal Electronic Health Records via Hierarchical Autoregressive Language Model
• Can segmentation models be trained with fully synthetically generated data?
• Evaluating the Clinical Realism of Synthetic Chest X-Rays Generated Using Progressively Growing GANs
• Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization
• Identifying Women with Mammographically-Occult Breast Cancer Leveraging GAN-Simulated Mammograms
• A Data Augmentation Pipeline to Generate Synthetic Labeled Datasets of 3D Echocardiography Images using a GAN
• High-resolution synthesis of high-density breast mammograms: Application to improved fairness in deep learning based mass detection
• Deep Learning based Multi-modal Computing with Feature Disentanglement for MRI Image Synthesis
• Data Augmentation for Seizure Prediction with Generative Diffusion Model
• The (de)biasing effect of GAN-based augmentation methods on skin lesion images
• MAM-E: Mammographic synthetic image generation with diffusion models
• TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network
• deepPERFECT: Novel Deep Learning CT Synthesis Method for Expeditious Pancreatic Cancer Radiotherapy
• A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion
• Improving dermatology classifiers across populations using images generated by large diffusion models
• The Swiss Army Knife for Image-to-Image Translation: Multi-Task Diffusion Models
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations
• Synthetic ECG Signal Generation Using Generative Neural Networks
• Augmenting medical image classifiers with synthetic data from latent diffusion models
• 3D Brain and Heart Volume Generative Models: A Survey
• MedDiff: Generating Electronic Health Records using Accelerated Denoising Diffusion Model
• Medical diffusion on a budget: Textual Inversion for medical image generation
• DDMM-Synth: A Denoising Diffusion Model for Cross-modal Medical Image Synthesis with Sparse-view Measurement Embedding
• Bi-parametric prostate MR image synthesis using pathology and sequence-conditioned stable diffusion
• Synthetic Health-related Longitudinal Data with Mixed-type Variables Generated using Diffusion Models
• A Comprehensive Survey on Generative Diffusion Models for Structured Data
• EHRDiff: Exploring Realistic EHR Synthesis with Diffusion Models
• DiffECG: A Versatile Probabilistic Diffusion Model for ECG Signals Synthesis
• Synthetic Data as Validation
• Downstream Fairness Caveats with Synthetic Healthcare Data
• Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs
• Towards Realistic Ultrasound Fetal Brain Imaging Synthesis
• Synthesizing Mixed-type Electronic Health Records using Diffusion Models
• Vision-Language Generative Model for View-Specific Chest X-ray Generation
• Towards Synthesizing Twelve-Lead Electrocardiograms from Two Asynchronous Leads
• Enabling Competitive Performance of Medical Imaging with Diffusion Model-generated Images without Privacy Leakage
• DP-MDM: Detail-Preserving MR Reconstruction via Multiple Diffusion Models
• LatentAugment: Data Augmentation via Guided Manipulation of GAN's Latent Space
• DiffBoost: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model
• ECGNet: A generative adversarial network (GAN) approach to the synthesis of 12-lead ECG signals from single lead inputs
• Subpopulation-Specific Synthetic EHR for Better Mortality Prediction
• TimeAutoDiff: A Unified Framework for Generation, Imputation, Forecasting, and Time-Varying Metadata Conditioning of Heterogeneous Time Series Tabular Data
• Evaluation of Synthetically Generated CT for use in Transcranial Focused Ultrasound Procedures
• PhysioGAN: Training High Fidelity Generative Model for Physiological Sensor Readings
• Method for Generating Synthetic Data Combining Chest Radiography Images with Tabular Clinical Information Using Dual Generative Models
• LungViT: Ensembling Cascade of Texture Sensitive Hierarchical Vision Transformers for Cross-Volume Chest CT Image-to-Image Translation
• A Data Augmentation Method and the Embedding Mechanism for Detection and Classification of Pulmonary Nodules on Small Samples
• Cyclic Generative Adversarial Networks With Congruent Image-Report Generation For Explainable Medical Image Analysis
• DACov: A Deeper Analysis of Data Augmentation on the Computed Tomography Segmentation Problem

Cited by in corpus (3)

• Methods for generating and evaluating synthetic longitudinal patient data: a systematic review
• Evaluation metrics for temporal preservation in synthetic longitudinal patient data
• Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances