Distribution Matching Losses Can Hallucinate Features in Medical Image Translation

Abstract

This paper discusses how distribution matching losses, such as those used in CycleGAN, when used to synthesize medical images can lead to mis-diagnosis of medical conditions. It seems appealing to use these new image synthesis methods for translating images from a source to a target domain because they can produce high quality images and some even do not require paired data. However, the basis of how these image translation models work is through matching the translation output to the distribution of the target domain. This can cause an issue when the data provided in the target domain has an over or under representation of some classes (e.g. healthy or sick). When the output of an algorithm is a transformed image there are uncertainties whether all known and unknown class labels have been preserved or changed. Therefore, we recommend that these translated images should not be used for direct interpretation (e.g. by doctors) because they may lead to misdiagnosis of patients based on hallucinated image features by an algorithm that matches a distribution. However there are many recent papers that seem as though this is the goal.

Published at Medical Image Computing & Computer Assisted Intervention (MICCAI 2018). An abstract is published at the Medical Imaging with Deep Learning Conference (MIDL 2018) as "How to Cure Cancer (in images) with Unpaired Image Translation"

References in corpus (6)

• Conditional Generative Adversarial Nets
• Adversarially Learned Inference
• Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss
• CycleGAN, a Master of Steganography
• Virtual PET Images from CT Data Using Deep Convolutional Networks: Initial Results
• Medical Image Synthesis with Context-Aware Generative Adversarial Networks

Cited by in corpus (28)

• Generative Adversarial Network in Medical Imaging: A Review
• COVID-19 Image Data Collection: Prospective Predictions Are the Future
• Deep learning-based transformation of the H&E stain into special stains
• SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth
• Learning with Known Operators reduces Maximum Training Error Bounds
• Melanoma Detection using Adversarial Training and Deep Transfer Learning
• Cross-modality (CT-MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets
• Pseudo-healthy synthesis with pathology disentanglement and adversarial learning
• PSIGAN: Joint probabilistic segmentation and image distribution matching for unpaired cross-modality adaptation based MRI segmentation
• Measuring Robustness in Deep Learning Based Compressive Sensing
• Multitask 3D CBCT-to-CT Translation and Organs-at-Risk Segmentation Using Physics-Based Data Augmentation
• Resolution enhancement and realistic speckle recovery with generative adversarial modeling of micro-optical coherence tomography
• High-resolution synthesis of high-density breast mammograms: Application to improved fairness in deep learning based mass detection
• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• Generating Diffusion MRI scalar maps from T1 weighted images using generative adversarial networks
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• Harmonization of diffusion MRI datasets with adaptive dictionary learning
• Generative adversarial networks and adversarial methods in biomedical image analysis
• Swap-Free Fat-Water Separation in Dixon MRI using Conditional Generative Adversarial Networks
• Early Screening of SARS-CoV-2 by Intelligent Analysis of X-Ray Images
• Galaxy Image Translation with Semi-supervised Noise-reconstructed Generative Adversarial Networks
• Comparison of Patch-Based Conditional Generative Adversarial Neural Net Models with Emphasis on Model Robustness for Use in Head and Neck Cases for MR-Only planning
• Empirical Bayesian Mixture Models for Medical Image Translation
• Image Quality Transfer Enhances Contrast and Resolution of Low-Field Brain MRI in African Paediatric Epilepsy Patients
• Synth-by-Reg (SbR): Contrastive learning for synthesis-based registration of paired images
• Deep Learning for Low-Field to High-Field MR: Image Quality Transfer with Probabilistic Decimation Simulator
• Optimizing transformations for contrastive learning in a differentiable framework
• MR-Contrast-Aware Image-to-Image Translations with Generative Adversarial Networks