Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Abstract

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?} To address this question, we considered 4 distinct medical imaging applications in 3 specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from 3 different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that (1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; (3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and (4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Factors of Transferability for a Generic ConvNet Representation

Cited by in corpus (68)

• CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection
• Transfer Learning with Deep Convolutional Neural Network (CNN) for Pneumonia Detection using Chest X-ray
• Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Machine Learning Techniques for Biomedical Image Segmentation: An Overview of Technical Aspects and Introduction to State-of-Art Applications
• Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images
• Automatic Colon Polyp Detection using Region based Deep CNN and Post Learning Approaches
• Skin Lesion Classification Using CNNs with Patch-Based Attention and Diagnosis-Guided Loss Weighting
• Quantifying the Carbon Emissions of Machine Learning
• Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation
• Abnormal Colon Polyp Image Synthesis Using Conditional Adversarial Networks for Improved Detection Performance
• Towards a Computed-Aided Diagnosis System in Colonoscopy: Automatic Polyp Segmentation Using Convolution Neural Networks
• Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?
• Deep learning trends for focal brain pathology segmentation in MRI
• COVID-DA: Deep Domain Adaptation from Typical Pneumonia to COVID-19
• Multi-task pre-training of deep neural networks for digital pathology
• Deep Transfer Learning Methods for Colon Cancer Classification in Confocal Laser Microscopy Images
• Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition
• Deep Learning-based Prediction of Key Performance Indicators for Electrical Machine
• Real-time automatic polyp detection in colonoscopy using feature enhancement module and spatiotemporal similarity correlation unit
• Identifying Women with Mammographically-Occult Breast Cancer Leveraging GAN-Simulated Mammograms
• Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes
• Ensembling Handcrafted Features with Deep Features: An Analytical Study for Classification of Routine Colon Cancer Histopathological Nuclei Images
• CVR-Net: A deep convolutional neural network for coronavirus recognition from chest radiography images
• Convolutional Neural Networks for Histopathology Image Classification: Training vs. Using Pre-Trained Networks
• From Patch to Image Segmentation using Fully Convolutional Networks -- Application to Retinal Images
• Iterative Multi-domain Regularized Deep Learning for Anatomical Structure Detection and Segmentation from Ultrasound Images
• Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation
• CSGAN: Cyclic-Synthesized Generative Adversarial Networks for Image-to-Image Transformation
• Surrogate Supervision for Medical Image Analysis: Effective Deep Learning From Limited Quantities of Labeled Data
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Exploring the Effect of Image Enhancement Techniques on COVID-19 Detection using Chest X-rays Images
• CNN-based Survival Model for Pancreatic Ductal Adenocarcinoma in Medical Imaging
• Multi-Stage Pathological Image Classification using Semantic Segmentation
• Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess
• AI-Skin : Skin Disease Recognition based on Self-learning and Wide Data Collection through a Closed Loop Framework
• TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning
• Pi-PE: A Pipeline for Pulmonary Embolism Detection using Sparsely Annotated 3D CT Images
• Fully automated analysis of muscle architecture from B-mode ultrasound images with deep learning
• Colorectal Cancer Outcome Prediction from H&E Whole Slide Images using Machine Learning and Automatically Inferred Phenotype Profiles
• Suggestive Annotation of Brain Tumour Images with Gradient-guided Sampling
• Mode Angular Degree Identification in Subgiant Stars with Convolutional Neural Networks based on Power Spectrum
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• DeepSperm: A robust and real-time bull sperm-cell detection in densely populated semen videos
• Advances and Challenges in Deep Lip Reading
• Fully Automatic Segmentation of Sublingual Veins from Retrained U-Net Model for Few Near Infrared Images
• Transfer Learning in 4D for Breast Cancer Diagnosis using Dynamic Contrast-Enhanced Magnetic Resonance Imaging
• Deep Learning-Based Automatic Detection of Poorly Positioned Mammograms to Minimize Patient Return Visits for Repeat Imaging: A Real-World Application
• Google Auto ML versus Apple Create ML for Histopathologic Cancer Diagnosis; Which Algorithms Are Better?
• Computer-Assisted Analysis of Biomedical Images
• Knowledge Transfer for Few-shot Segmentation of Novel White Matter Tracts
• Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning
• Automated Pancreas Segmentation Using Multi-institutional Collaborative Deep Learning
• Vanishing Nodes: Another Phenomenon That Makes Training Deep Neural Networks Difficult
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications
• Canoe : A System for Collaborative Learning for Neural Nets
• Spatio-Temporal Dual-Stream Neural Network for Sequential Whole-Body PET Segmentation
• A weakly supervised registration-based framework for prostate segmentation via the combination of statistical shape model and CNN
• Learning Efficient, Explainable and Discriminative Representations for Pulmonary Nodules Classification
• Machine Learning on Biomedical Images: Interactive Learning, Transfer Learning, Class Imbalance, and Beyond
• Informative sample generation using class aware generative adversarial networks for classification of chest Xrays
• On evaluating CNN representations for low resource medical image classification
• Deep Learning in Ultrasound Elastography Imaging
• ATSO: Asynchronous Teacher-Student Optimization for Semi-Supervised Medical Image Segmentation
• Development of the algorithm for differentiating bone metastases and trauma of the ribs in bone scintigraphy and demonstration of visual evidence of the algorithm -- Using only anterior bone scan view of thorax