Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Abstract

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?} To address this question, we considered 4 distinct medical imaging applications in 3 specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from 3 different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that (1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; (3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and (4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Brain Tumor Segmentation with Deep Neural Networks
• Going Deeper with Convolutions
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Deep convolutional networks for pancreas segmentation in CT imaging
• Factors of Transferability for a Generic ConvNet Representation

Cited by in corpus (192)

• A Survey on Deep Learning in Medical Image Analysis
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Can AI help in screening Viral and COVID-19 pneumonia?
• Recent advances and clinical applications of deep learning in medical image analysis
• Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning
• CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection
• Deep neural network models for computational histopathology: A survey
• BACH: Grand Challenge on Breast Cancer Histology Images
• Single-chip photonic deep neural network for instantaneous image classification
• Reliable Tuberculosis Detection using Chest X-ray with Deep Learning, Segmentation and Visualization
• Transfer Learning with Deep Convolutional Neural Network (CNN) for Pneumonia Detection using Chest X-ray
• A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis
• Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
• Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers
• Med3D: Transfer Learning for 3D Medical Image Analysis
• A Comprehensive Analysis of Deep Regression
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• Models Genesis
• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Machine Learning Techniques for Biomedical Image Segmentation: An Overview of Technical Aspects and Introduction to State-of-Art Applications
• Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation
• Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images
• Modality specific U-Net variants for biomedical image segmentation: A survey
• The impact of patient clinical information on automated skin cancer detection
• Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer
• Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training
• Automatic Colon Polyp Detection using Region based Deep CNN and Post Learning Approaches
• Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data
• SD-CNN: a Shallow-Deep CNN for Improved Breast Cancer Diagnosis
• Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications
• Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation
• Skin Lesion Classification Using CNNs with Patch-Based Attention and Diagnosis-Guided Loss Weighting
• Automatic Pulmonary Nodule Detection in CT Scans Using Convolutional Neural Networks Based on Maximum Intensity Projection
• Review of multi-fidelity models
• Computer-aided diagnosis in histopathological images of the endometrium using a convolutional neural network and attention mechanisms
• Evaluation of machine learning algorithms for Health and Wellness applications: a tutorial
• MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning
• Deep Learning for Computational Cytology: A Survey
• Quantifying the Carbon Emissions of Machine Learning
• Automatic Plaque Detection in IVOCT Pullbacks Using Convolutional Neural Networks
• COV-ECGNET: COVID-19 detection using ECG trace images with deep convolutional neural network
• Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation
• FedDPGAN: Federated Differentially Private Generative Adversarial Networks Framework for the Detection of COVID-19 Pneumonia
• Image computing for fibre-bundle endomicroscopy: A review
• Comparison of fine-tuning strategies for transfer learning in medical image classification
• Abnormal Colon Polyp Image Synthesis Using Conditional Adversarial Networks for Improved Detection Performance
• A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
• Transfer learning for galaxy morphology from one survey to another
• A Systematic Review on Model Watermarking for Neural Networks
• Self-supervised deep convolutional neural network for chest X-ray classification
• A review of federated learning in renewable energy applications: Potential, challenges, and future directions
• A Comparison of 1-D and 2-D Deep Convolutional Neural Networks in ECG Classification
• Building medical image classifiers with very limited data using segmentation networks
• An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification
• Cats or CAT scans: transfer learning from natural or medical image source datasets?
• Towards a Computed-Aided Diagnosis System in Colonoscopy: Automatic Polyp Segmentation Using Convolution Neural Networks
• Learning from models beyond fine-tuning
• Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?
• Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
• Evolving the pulmonary nodules diagnosis from classical approaches to deep learning aided decision support: three decades development course and future prospect
• Deep learning trends for focal brain pathology segmentation in MRI
• DC-AL GAN: Pseudoprogression and True Tumor Progression of Glioblastoma Multiform Image Classification Based on DCGAN and AlexNet
• COVID-DA: Deep Domain Adaptation from Typical Pneumonia to COVID-19
• Multi-task pre-training of deep neural networks for digital pathology
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Image Based Fashion Product Recommendation with Deep Learning
• EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks
• Forecasting Megaelectron-Volt Electrons inside Earth's Outer Radiation Belt: PreMevE 2.0 Based on Supervised Machine Learning Algorithms
• Multiscale CNN based Deep Metric Learning for Bioacoustic Classification: Overcoming Training Data Scarcity Using Dynamic Triplet Loss
• DeepBeam: Deep Waveform Learning for Coordination-Free Beam Management in mmWave Networks
• Skin cancer detection based on deep learning and entropy to detect outlier samples
• Decision and Feature Level Fusion of Deep Features Extracted from Public COVID-19 Data-sets
• Vertex and Energy Reconstruction in JUNO with Machine Learning Methods
• Deep Learning based HEp-2 Image Classification: A Comprehensive Review
• Computer-Aided Diagnosis of Label-Free 3-D Optical Coherence Microscopy Images of Human Cervical Tissue
• Industrial object, machine part and defect recognition towards fully automated industrial monitoring employing deep learning. The case of multilevel VGG19
• Deep Transfer Learning Methods for Colon Cancer Classification in Confocal Laser Microscopy Images
• Convolutional Neural Networks: Ensemble Modeling, Fine-Tuning and Unsupervised Semantic Localization for Intraoperative CLE Images
• Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning
• SpotTune: Transfer Learning through Adaptive Fine-tuning
• Convolutional Sparse Kernel Network for Unsupervised Medical Image Analysis
• Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition
• Deep Learning with Cinematic Rendering: Fine-Tuning Deep Neural Networks Using Photorealistic Medical Images
• Deep Learning-based Prediction of Key Performance Indicators for Electrical Machine
• Real-time automatic polyp detection in colonoscopy using feature enhancement module and spatiotemporal similarity correlation unit
• Identifying Women with Mammographically-Occult Breast Cancer Leveraging GAN-Simulated Mammograms
• Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes
• Medical Transformer: Universal Brain Encoder for 3D MRI Analysis
• A Survey on Deep Learning of Small Sample in Biomedical Image Analysis
• An Unsupervised Deep Learning Method for Multi-coil Cine MRI
• Ensembling Handcrafted Features with Deep Features: An Analytical Study for Classification of Routine Colon Cancer Histopathological Nuclei Images
• Thickened 2D Networks for Efficient 3D Medical Image Segmentation
• CVR-Net: A deep convolutional neural network for coronavirus recognition from chest radiography images
• CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters
• How Much Off-The-Shelf Knowledge Is Transferable From Natural Images To Pathology Images?
• Convolutional Neural Networks for Histopathology Image Classification: Training vs. Using Pre-Trained Networks
• From Patch to Image Segmentation using Fully Convolutional Networks -- Application to Retinal Images
• Left Ventricle Segmentation and Volume Estimation on Cardiac MRI using Deep Learning
• DeepHealth: Review and challenges of artificial intelligence in health informatics
• In-Bed Human Pose Estimation from Unseen and Privacy-Preserving Image Domains
• Learning Models of Individual Behavior in Chess
• Deep Learning Analysis of Cardiac MRI in Legacy Datasets: Multi-Ethnic Study of Atherosclerosis
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Iterative Multi-domain Regularized Deep Learning for Anatomical Structure Detection and Segmentation from Ultrasound Images
• A Deep Dive into Understanding Tumor Foci Classification using Multiparametric MRI Based on Convolutional Neural Network
• FlowMur: A Stealthy and Practical Audio Backdoor Attack with Limited Knowledge
• CCL: Continual Contrastive Learning for LiDAR Place Recognition
• AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients
• Simple U-net Based Synthetic Polyp Image Generation: Polyp to Negative and Negative to Polyp
• Exploring Optimal Deep Learning Models for Image-based Malware Variant Classification
• Automated Thalamic Nuclei Segmentation Using Multi-Planar Cascaded Convolutional Neural Networks
• Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation
• Optimizing Quantum Convolutional Neural Network Architectures for Arbitrary Data Dimension
• Computer Aided Detection for Pulmonary Embolism Challenge (CAD-PE)
• Surrogate Supervision for Medical Image Analysis: Effective Deep Learning From Limited Quantities of Labeled Data
• CSGAN: Cyclic-Synthesized Generative Adversarial Networks for Image-to-Image Transformation
• Feature Representation Analysis of Deep Convolutional Neural Network using Two-stage Feature Transfer -An Application for Diffuse Lung Disease Classification-
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Momentum Residual Neural Networks
• Combining Image Features and Patient Metadata to Enhance Transfer Learning
• Optimize transfer learning for lung diseases in bronchoscopy using a new concept: sequential fine-tuning
• Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization
• Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World
• Exploring the Effect of Image Enhancement Techniques on COVID-19 Detection using Chest X-rays Images
• Does Non-COVID19 Lung Lesion Help? Investigating Transferability in COVID-19 CT Image Segmentation
• DFPENet-geology: A Deep Learning Framework for High Precision Recognition and Segmentation of Co-seismic Landslides
• Multi-Stage Pathological Image Classification using Semantic Segmentation
• Improving Robustness and Reliability in Medical Image Classification with Latent-Guided Diffusion and Nested-Ensembles
• CNN-based Survival Model for Pancreatic Ductal Adenocarcinoma in Medical Imaging
• Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess
• AI-Skin : Skin Disease Recognition based on Self-learning and Wide Data Collection through a Closed Loop Framework
• Pi-PE: A Pipeline for Pulmonary Embolism Detection using Sparsely Annotated 3D CT Images
• Elastic Boundary Projection for 3D Medical Image Segmentation
• TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning
• Colorectal Cancer Outcome Prediction from H&E Whole Slide Images using Machine Learning and Automatically Inferred Phenotype Profiles
• Fully automated analysis of muscle architecture from B-mode ultrasound images with deep learning
• Suggestive Annotation of Brain Tumour Images with Gradient-guided Sampling
• EfficientNet Algorithm for Classification of Different Types of Cancer
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• Intentional Deep Overfit Learning (IDOL): A Novel Deep Learning Strategy for Adaptive Radiation Therapy
• On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study
• A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers
• Bio-inspired fine-tuning for selective transfer learning in image classification
• Mode Angular Degree Identification in Subgiant Stars with Convolutional Neural Networks based on Power Spectrum
• Evaluating Convolutional Neural Networks for COVID-19 classification in chest X-ray images
• DeepSperm: A robust and real-time bull sperm-cell detection in densely populated semen videos
• Transfer Learning from Partial Annotations for Whole Brain Segmentation
• Towards Efficient Convolutional Neural Network for Domain-Specific Applications on FPGA
• Fully Automatic Segmentation of Sublingual Veins from Retrained U-Net Model for Few Near Infrared Images
• Deep Learning-Based Automatic Detection of Poorly Positioned Mammograms to Minimize Patient Return Visits for Repeat Imaging: A Real-World Application
• Intensity augmentation for domain transfer of whole breast segmentation in MRI
• Analysis of the Two-Step Heterogeneous Transfer Learning for Laryngeal Blood Vessel Classification: Issue and Improvement
• Transfer Learning in 4D for Breast Cancer Diagnosis using Dynamic Contrast-Enhanced Magnetic Resonance Imaging
• Self-Adaptive Transfer Learning for Multicenter Glaucoma Classification in Fundus Retina Images
• Classification of Chest Diseases using Wavelet Transforms and Transfer Learning
• Google Auto ML versus Apple Create ML for Histopathologic Cancer Diagnosis; Which Algorithms Are Better?
• A Deep Learning-Based Unified Framework for Red Lesions Detection on Retinal Fundus Images
• Advances and Challenges in Deep Lip Reading
• 3D medical image segmentation with labeled and unlabeled data using autoencoders at the example of liver segmentation in CT images
• Computer-Assisted Analysis of Biomedical Images
• Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Vanishing Nodes: Another Phenomenon That Makes Training Deep Neural Networks Difficult
• Knowledge Transfer for Few-shot Segmentation of Novel White Matter Tracts
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• Towards to Robust and Generalized Medical Image Segmentation Framework
• ImageNet Pre-training also Transfers Non-Robustness
• Automated Pancreas Segmentation Using Multi-institutional Collaborative Deep Learning
• Informative sample generation using class aware generative adversarial networks for classification of chest Xrays
• Knee menisci segmentation and relaxometry of 3D ultrashort echo time (UTE) cones MR imaging using attention U-Net with transfer learning
• Development of the algorithm for differentiating bone metastases and trauma of the ribs in bone scintigraphy and demonstration of visual evidence of the algorithm -- Using only anterior bone scan view of thorax
• Canoe : A System for Collaborative Learning for Neural Nets
• Learning Efficient, Explainable and Discriminative Representations for Pulmonary Nodules Classification
• Spatio-Temporal Dual-Stream Neural Network for Sequential Whole-Body PET Segmentation
• SUSAN: Segment Unannotated image Structure using Adversarial Network
• ATSO: Asynchronous Teacher-Student Optimization for Semi-Supervised Medical Image Segmentation
• Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications
• A weakly supervised registration-based framework for prostate segmentation via the combination of statistical shape model and CNN
• Learning to Harmonize Cross-vendor X-ray Images by Non-linear Image Dynamics Correction
• A Dataset of Laryngeal Endoscopic Images with Comparative Study on Convolution Neural Network Based Semantic Segmentation
• How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring
• Tissue characterization based on the analysis on i3DUS data for diagnosis support in neurosurgery
• Estimation of Acetabular Version from Anteroposterior Pelvic Radiograph Employing Deep Learning
• Machine Learning on Biomedical Images: Interactive Learning, Transfer Learning, Class Imbalance, and Beyond
• Enhanced Transfer Learning Through Medical Imaging and Patient Demographic Data Fusion
• Adaptive Transfer Learning: a simple but effective transfer learning
• On evaluating CNN representations for low resource medical image classification
• Deep Learning in Ultrasound Elastography Imaging
• A Novel Deep ML Architecture by Integrating Visual Simultaneous Localization and Mapping (vSLAM) into Mask R-CNN for Real-time Surgical Video Analysis
• Unsupervised Domain-adaptive Hash for Networks