EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision
arXiv:2007.05597
Abstract
Deep generative models have enabled the automated synthesis of high-quality data for diverse applications. However, the most effective generative models are specialized to data from a single domain (e.g., images or text). Real-world applications such as healthcare require multi-modal data from multiple domains (e.g., both images and corresponding text), which are difficult to acquire due to limited availability and privacy concerns and are much harder to synthesize. To tackle this joint synthesis challenge, we propose an End-to-end MultImodal X-ray genERative model (EMIXER) for jointly synthesizing x-ray images and corresponding free-text reports, all conditional on diagnosis labels. EMIXER is an conditional generative adversarial model by 1) generating an image based on a label, 2) encoding the image to a hidden embedding, 3) producing the corresponding text via a hierarchical decoder from the image embedding, and 4) a joint discriminator for assessing both the image and the corresponding text. EMIXER also enables self-supervision to leverage vast amount of unlabeled data. Extensive experiments with real X-ray reports data illustrate how data augmentation using synthesized multimodal samples can improve the performance of a variety of supervised tasks including COVID-19 X-ray classification with very limited samples. The quality of generated images and reports are also confirmed by radiologists. We quantitatively show that EMIXER generated synthetic datasets can augment X-ray image classification, report generation models to achieve 5.94% and 6.9% improvement on models trained only on real data samples. Taken together, our results highlight the promise of state of generative models to advance clinical machine learning.
References in corpus (19)
- Auto-Encoding Variational Bayes
- Conditional Generative Adversarial Nets
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Self-Attention Generative Adversarial Networks
- Conditional Image Generation with PixelCNN Decoders
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
- COVID-19 Image Data Collection
- Data Augmentation Generative Adversarial Networks
- Density estimation using Real NVP
- Semantic Segmentation using Adversarial Networks
- Generating Multi-label Discrete Patient Records using Generative Adversarial Networks
- MaskGAN: Better Text Generation via Filling in the______
- Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
- Clinically Accurate Chest X-Ray Report Generation
- Quick and Easy Time Series Generation with Established Image-based GANs
- JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets