Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation

Abstract

The medical imaging literature has witnessed remarkable progress in high-performing segmentation models based on convolutional neural networks. Despite the new performance highs, the recent advanced segmentation models still require large, representative, and high quality annotated datasets. However, rarely do we have a perfect training dataset, particularly in the field of medical imaging, where data and annotations are both expensive to acquire. Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations. In this article, we provide a detailed review of the solutions above, summarizing both the technical novelties and empirical results. We further compare the benefits and requirements of the surveyed methodologies and provide our recommended solutions. We hope this survey article increases the community awareness of the techniques that are available to handle imperfect medical image segmentation datasets.

arXiv commentAccepted for publication in the journal of Medical Image Analysis

References in corpus (75)

• A Survey on Deep Learning in Medical Image Analysis
• mixup: Beyond Empirical Risk Minimization
• Attention U-Net: Learning Where to Look for the Pancreas
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Conditional Random Fields as Recurrent Neural Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Generative Adversarial Network in Medical Imaging: A Review
• Understanding deep learning requires rethinking generalization
• Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning
• Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation
• Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields
• A Survey on Multi-Task Learning
• DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
• 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation
• DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy
• Patch-based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation
• Multimodal Unsupervised Image-to-Image Translation
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• Fully Connected Deep Structured Networks
• Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation
• Semi-Supervised Medical Image Segmentation via Learning Consistency under Transformations
• Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules
• DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation
• Semi-Supervised Deep Learning for Fully Convolutional Networks
• Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learning
• Synthetic Medical Images from Dual Generative Adversarial Networks
• XLSor: A Robust and Accurate Lung Segmentor on Chest X-Rays Using Criss-Cross Attention and Customized Radiorealistic Abnormalities Generation
• 3D RoI-aware U-Net for Accurate and Efficient Colorectal Tumor Segmentation
• Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation
• Three Dimensional Fluorescence Microscopy Image Synthesis and Segmentation
• Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks
• Cost-Effective Active Learning for Melanoma Segmentation
• Unsupervised Multi-modal Style Transfer for Cardiac MR Segmentation
• Interactive segmentation of medical images through fully convolutional neural networks
• Data augmentation using learned transformations for one-shot medical image segmentation
• Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network
• Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation
• PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network with a Benchmark at Cross-modality Cardiac Segmentation
• Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
• Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation
• A Survey on Deep Learning of Small Sample in Biomedical Image Analysis
• Self-Supervised Similarity Learning for Digital Pathology
• Transformation Consistent Self-ensembling Model for Semi-supervised Medical Image Segmentation
• Active Learning for Segmentation by Optimizing Content Information for Maximal Entropy
• Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction
• DeepAtlas: Joint Semi-Supervised Learning of Image Registration and Segmentation
• CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation
• Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation
• A sparse annotation strategy based on attention-guided active learning for 3D medical image segmentation
• Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training
• Conditional Random Fields as Recurrent Neural Networks for 3D Medical Imaging Segmentation
• Self-Supervised Learning for Spinal MRIs
• Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation
• Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation
• Multi-Task Attention-Based Semi-Supervised Learning for Medical Image Segmentation
• Accurate Weakly Supervised Deep Lesion Segmentation on CT Scans: Self-Paced 3D Mask Generation from RECIST
• Surrogate Supervision for Medical Image Analysis: Effective Deep Learning From Limited Quantities of Labeled Data
• Semi-Supervised Brain Lesion Segmentation with an Adapted Mean Teacher Model
• Interactive Medical Image Segmentation via Point-Based Interaction and Sequential Patch Learning
• Transfer Learning with Edge Attention for Prostate MRI Segmentation
• Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection
• Learning to Segment Medical Images with Scribble-Supervision Alone
• An End-to-end Approach to Semantic Segmentation with 3D CNN and Posterior-CRF in Medical Images
• Recurrent Aggregation Learning for Multi-View Echocardiographic Sequences Segmentation
• Neural Style Transfer Improves 3D Cardiovascular MR Image Segmentation on Inconsistent Data
• Self-Learning to Detect and Segment Cysts in Lung CT Images without Manual Annotation
• Pick-and-Learn: Automatic Quality Evaluation for Noisy-Labeled Image Segmentation
• Unsupervised Domain Adaptation for Automatic Estimation of Cardiothoracic Ratio

Cited by in corpus (61)

• Recent advances and clinical applications of deep learning in medical image analysis
• A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis
• Annotation-efficient deep learning for automatic medical image segmentation
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions
• BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset
• Reinventing 2D Convolutions for 3D Images
• Review: Deep Learning in Electron Microscopy
• Fully Automated Multi-Organ Segmentation in Abdominal Magnetic Resonance Imaging with Deep Neural Networks
• Semi-Supervised Segmentation of Radiation-Induced Pulmonary Fibrosis from Lung CT Scans with Multi-Scale Guided Dense Attention
• UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation
• Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
• A Unified Framework for Generalized Low-Shot Medical Image Segmentation with Scarce Data
• UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation
• medigan: a Python library of pretrained generative models for medical image synthesis
• MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation
• Hierarchical Consistency Regularized Mean Teacher for Semi-supervised 3D Left Atrium Segmentation
• Constrained domain adaptation for Image segmentation
• CarveMix: A Simple Data Augmentation Method for Brain Lesion Segmentation
• Voice-assisted Image Labelling for Endoscopic Ultrasound Classification using Neural Networks
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images
• AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?
• Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?
• Unsupervised Medical Image Segmentation with Adversarial Networks: From Edge Diagrams to Segmentation Maps
• FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging
• Emergency Department Decision Support using Clinical Pseudo-notes
• Terabyte-scale supervised 3D training and benchmarking dataset of the mouse kidney
• Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT
• Dual-Task Mutual Learning for Semi-Supervised Medical Image Segmentation
• Self-Supervised Learning for 3D Medical Image Analysis using 3D SimCLR and Monte Carlo Dropout
• Deep Mining External Imperfect Data for Chest X-ray Disease Screening
• Universal Lesion Detection by Learning from Multiple Heterogeneously Labeled Datasets
• AGMB-Transformer: Anatomy-Guided Multi-Branch Transformer Network for Automated Evaluation of Root Canal Therapy
• Extreme Consistency: Overcoming Annotation Scarcity and Domain Shifts
• Critical Assessment of Transfer Learning for Medical Image Segmentation with Fully Convolutional Neural Networks
• Exploiting Shared Knowledge from Non-COVID Lesions for Annotation-Efficient COVID-19 CT Lung Infection Segmentation
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Contour Transformer Network for One-shot Segmentation of Anatomical Structures
• A novel open-source ultrasound dataset with deep learning benchmarks for spinal cord injury localization and anatomical segmentation
• TricycleGAN: Unsupervised Image Synthesis and Segmentation Based on Shape Priors
• A Location-Sensitive Local Prototype Network for Few-Shot Medical Image Segmentation
• Tips and Tricks to Improve CNN-based Chest X-ray Diagnosis: A Survey
• Look at here : Utilizing supervision to attend subtle key regions
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Generalizing electrocardiogram delineation -- Training convolutional neural networks with synthetic data augmentation
• Deep Structure Learning using Feature Extraction in Trained Projection Space
• Classifying Breast Histopathology Images with a Ductal Instance-Oriented Pipeline
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Deep Learning for Earth Image Segmentation based on Imperfect Polyline Labels with Annotation Errors
• Hetero-Modal Learning and Expansive Consistency Constraints for Semi-Supervised Detection from Multi-Sequence Data
• Automated Multi-Channel Segmentation for the 4D Myocardial Velocity Mapping Cardiac MR
• A Research Agenda on Pediatric Chest X-Ray: Is Deep Learning Still in Childhood?
• ErrorNet: Learning error representations from limited data to improve vascular segmentation
• OXnet: Omni-supervised Thoracic Disease Detection from Chest X-rays
• One-shot Weakly-Supervised Segmentation in Medical Images
• Vessel-CAPTCHA: an efficient learning framework for vessel annotation and segmentation
• Dual Shape Guided Segmentation Network for Organs-at-Risk in Head and Neck CT Images
• Unpaired Single-Image Depth Synthesis with cycle-consistent Wasserstein GANs
• The Power of Proxy Data and Proxy Networks for Hyper-Parameter Optimization in Medical Image Segmentation