Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning

Abstract

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine-tuning. We applied this framework to two applications: 2D segmentation of multiple organs from fetal MR slices, where only two types of these organs were annotated for training; and 3D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only tumor cores in one MR sequence were annotated for training. Experimental results show that 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine-tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

arXiv comment11 pages, 11 figures

References in corpus (13)

• A Survey on Deep Learning in Medical Image Analysis
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Fully Convolutional Networks for Semantic Segmentation
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Transductive Multi-view Zero-Shot Learning
• Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks
• DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
• On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task
• DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
• Scalable multimodal convolutional networks for brain tumour segmentation

Cited by in corpus (84)

• Segment Anything in Medical Images
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• U-Net and its variants for medical image segmentation: theory and applications
• Medical Image Segmentation Using Deep Learning: A Survey
• CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation
• A Survey of Human-in-the-loop for Machine Learning
• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis
• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration
• Extracting possibly representative COVID-19 Biomarkers from X-Ray images with Deep Learning approach and image data related to Pulmonary Diseases
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation
• Automatic Brain Tumor Segmentation using Convolutional Neural Networks with Test-Time Augmentation
• MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning
• MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images
• CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
• Crossbar-Net: A Novel Convolutional Network for Kidney Tumor Segmentation in CT Images
• Bi-directional Dermoscopic Feature Learning and Multi-scale Consistent Decision Fusion for Skin Lesion Segmentation
• Qutrit-inspired Fully Self-supervised Shallow Quantum Learning Network for Brain Tumor Segmentation
• PA-ResSeg: A Phase Attention Residual Network for Liver Tumor Segmentation from Multi-phase CT Images
• RetiFluidNet: A Self-Adaptive and Multi-Attention Deep Convolutional Network for Retinal OCT Fluid Segmentation
• Manual segmentation versus semi-automated segmentation for quantifying vestibular schwannoma volume on MRI
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Gaussian Dynamic Convolution for Efficient Single-Image Segmentation
• Quantifying Uncertainty in Discrete-Continuous and Skewed Data with Bayesian Deep Learning
• Deep Learning Based 3D Segmentation: A Survey
• A Convex Geodesic Selective Model for Image Segmentation
• Joint registration and synthesis using a probabilistic model for alignment of MRI and histological sections
• Annotation-Efficient Learning for Medical Image Segmentation based on Noisy Pseudo Labels and Adversarial Learning
• PIG-Net: Inception based Deep Learning Architecture for 3D Point Cloud Segmentation
• Uncertainty-Guided Efficient Interactive Refinement of Fetal Brain Segmentation from Stacks of MRI Slices
• DeepEdit: Deep Editable Learning for Interactive Segmentation of 3D Medical Images
• Scribble-based Domain Adaptation via Co-segmentation
• Hyper-Fusion Network for Semi-Automatic Segmentation of Skin Lesions
• Inter Extreme Points Geodesics for End-to-End Weakly Supervised Image Segmentation
• Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalising neural network
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• Weakly supervised segmentation from extreme points
• How Will It Drape Like? Capturing Fabric Mechanics from Depth Images
• Generating 3D Bio-Printable Patches Using Wound Segmentation and Reconstruction to Treat Diabetic Foot Ulcers
• Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues
• DINs: Deep Interactive Networks for Neurofibroma Segmentation in Neurofibromatosis Type 1 on Whole-Body MRI
• NuClick: A Deep Learning Framework for Interactive Segmentation of Microscopy Images
• Uncertainty-based graph convolutional networks for organ segmentation refinement
• Decentralized Machine Learning for Intelligent Health Care Systems on the Computing Continuum
• Technical Understanding from IML Hands-on Experience: A Study through a Public Event for Science Museum Visitors
• Adaptive Multi-scale Online Likelihood Network for AI-assisted Interactive Segmentation
• Deep Learning in Multi-organ Segmentation
• Interactive Segmentation via Deep Learning and B-Spline Explicit Active Surfaces
• Automatic Segmentation of Organs-at-Risk from Head-and-Neck CT using Separable Convolutional Neural Network with Hard-Region-Weighted Loss
• A decentralized aggregation mechanism for training deep learning models using smart contract system for bank loan prediction
• LAFEAT: Piercing Through Adversarial Defenses with Latent Features
• Interactive Medical Image Segmentation with Self-Adaptive Confidence Calibration
• Fast and Scalable Image Search For Histology
• Learning To Pay Attention To Mistakes
• Interactive user interface based on Convolutional Auto-encoders for annotating CT-scans
• Look at here : Utilizing supervision to attend subtle key regions
• CMB delensing with deep learning
• Iterative Interaction Training for Segmentation Editing Networks
• OOD-SEG: Exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations
• SAMSA: Segment Anything Model Enhanced with Spectral Angles for Hyperspectral Interactive Medical Image Segmentation
• Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation
• A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images
• HAISTA-NET: Human Assisted Instance Segmentation Through Attention
• Computer-Assisted Analysis of Biomedical Images
• MitoVis: A Visually-guided Interactive Intelligent System for Neuronal Mitochondria Analysis
• FetusMap: Fetal Pose Estimation in 3D Ultrasound
• NFCNN: Toward a Noise Fusion Convolutional Neural Network for Image Denoising
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Radiotherapy Target Contouring with Convolutional Gated Graph Neural Network
• Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning
• An Alternative Practice of Tropical Convolution to Traditional Convolutional Neural Networks
• Label tree semantic losses for rich multi-class medical image segmentation
• Localized Interactive Instance Segmentation
• Scale-aware multi-level guidance for interactive instance segmentation
• Dual Shape Guided Segmentation Network for Organs-at-Risk in Head and Neck CT Images
• Vessel-CAPTCHA: an efficient learning framework for vessel annotation and segmentation
• CPNet: Cycle Prototype Network for Weakly-supervised 3D Renal Compartments Segmentation on CT Images
• Patient-specific fine-tuning of CNNs for follow-up lesion quantification
• A Dataset of Laryngeal Endoscopic Images with Comparative Study on Convolution Neural Network Based Semantic Segmentation
• Interactive Radiotherapy Target Delineation with 3D-Fused Context Propagation
• CleftNet: Augmented Deep Learning for Synaptic Cleft Detection from Brain Electron Microscopy
• A Research Agenda on Pediatric Chest X-Ray: Is Deep Learning Still in Childhood?
• W-Net: A Two-Stage Convolutional Network for Nucleus Detection in Histopathology Image