Transfusion: Understanding Transfer Learning for Medical Imaging

Abstract

Transfer learning from natural image datasets, particularly ImageNet, using standard large models and corresponding pretrained weights has become a de-facto method for deep learning applications to medical imaging. However, there are fundamental differences in data sizes, features and task specifications between natural image classification and the target medical tasks, and there is little understanding of the effects of transfer. In this paper, we explore properties of transfer learning for medical imaging. A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures. Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse. We isolate where useful feature reuse occurs, and outline the implications for more efficient model exploration. We also explore feature independent benefits of transfer arising from weight scalings.

NeurIPS 2019

References in corpus (13)

• How transferable are features in deep neural networks?
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Deep Learning for Identifying Metastatic Breast Cancer
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Similarity of Neural Network Representations Revisited
• What makes ImageNet good for transfer learning?
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Are All Layers Created Equal?
• Rethinking ImageNet Pre-training
• Domain Adaptive Transfer Learning with Specialist Models
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

Cited by in corpus (74)

• TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning
• A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis
• DeepCervix: A Deep Learning-based Framework for the Classification of Cervical Cells Using Hybrid Deep Feature Fusion Techniques
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
• IL-MCAM: An interactive learning and multi-channel attention mechanism-based weakly supervised colorectal histopathology image classification approach
• Do Vision Transformers See Like Convolutional Neural Networks?
• Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Learning With Context Feedback Loop for Robust Medical Image Segmentation
• MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models
• Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification
• A Closer Look at Domain Shift for Deep Learning in Histopathology
• Intra-Domain Task-Adaptive Transfer Learning to Determine Acute Ischemic Stroke Onset Time
• Machine Learning (In) Security: A Stream of Problems
• Exploring the Limits of Large Scale Pre-training
• GraphXCOVID: Explainable Deep Graph Diffusion Pseudo-Labelling for Identifying COVID-19 on Chest X-rays
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• Probing transfer learning with a model of synthetic correlated datasets
• MUSCLE: Multi-task Self-supervised Continual Learning to Pre-train Deep Models for X-ray Images of Multiple Body Parts
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy
• Learning Models of Individual Behavior in Chess
• Predicting Knee Osteoarthritis Progression from Structural MRI using Deep Learning
• Multi-structure bone segmentation in pediatric MR images with combined regularization from shape priors and adversarial network
• Evaluating histopathology transfer learning with ChampKit
• MedNet: Pre-trained Convolutional Neural Network Model for the Medical Imaging Tasks
• Few-shot learning for COVID-19 Chest X-Ray Classification with Imbalanced Data: An Inter vs. Intra Domain Study
• How Will It Drape Like? Capturing Fabric Mechanics from Depth Images
• ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning
• Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruning
• Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
• Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs
• Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries
• Does Non-COVID19 Lung Lesion Help? Investigating Transferability in COVID-19 CT Image Segmentation
• Transfer Learning Study of Motion Transformer-based Trajectory Predictions
• ICodeNet -- A Hierarchical Neural Network Approach for Source Code Author Identification
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Critical Assessment of Transfer Learning for Medical Image Segmentation with Fully Convolutional Neural Networks
• Learning Optical Flow for Fast MRI Reconstruction
• How Fine-Tuning Allows for Effective Meta-Learning
• Self-transfer learning via patches: A prostate cancer triage approach based on bi-parametric MRI
• Robust and Explainable Framework to Address Data Scarcity in Diagnostic Imaging
• Interpretable and synergistic deep learning for visual explanation and statistical estimations of segmentation of disease features from medical images
• Less is More: Selective Reduction of CT Data for Self-Supervised Pre-Training of Deep Learning Models with Contrastive Learning Improves Downstream Classification Performance
• Adversarial Training Helps Transfer Learning via Better Representations
• Towards Label-Free 3D Segmentation of Optical Coherence Tomography Images of the Optic Nerve Head Using Deep Learning
• Hybrid Deep Learning Model for epileptic seizure classification by using 1D-CNN with multi-head attention mechanism
• Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning
• Prototypical Cross-domain Knowledge Transfer for Cervical Dysplasia Visual Inspection
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Revisiting Mid-Level Patterns for Cross-Domain Few-Shot Recognition
• Parameter Prediction for Unseen Deep Architectures
• Generic Neural Architecture Search via Regression
• Deepfake Image Generation for Improved Brain Tumor Segmentation
• Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation
• Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum
• BMFT: Achieving Fairness via Bias-based Weight Masking Fine-tuning
• Revisiting Hidden Representations in Transfer Learning for Medical Imaging
• HistoTransfer: Understanding Transfer Learning for Histopathology
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Improving Interpretability for Computer-aided Diagnosis tools on Whole Slide Imaging with Multiple Instance Learning and Gradient-based Explanations
• Optimal Transfer Learning Model for Binary Classification of Funduscopic Images through Simple Heuristics
• Conditional Deep Convolutional Neural Networks for Improving the Automated Screening of Histopathological Images
• A Comparison of Methods for Neural Network Aggregation
• NeuroView: Explainable Deep Network Decision Making
• Requirement analysis for an artificial intelligence model for the diagnosis of the COVID-19 from chest X-ray data
• Deep Learning on SAR Imagery: Transfer Learning Versus Randomly Initialized Weights