Supervised Transfer Learning at Scale for Medical Imaging

Abstract

Transfer learning is a standard technique to improve performance on tasks with limited data. However, for medical imaging, the value of transfer learning is less clear. This is likely due to the large domain mismatch between the usual natural-image pre-training (e.g. ImageNet) and medical images. However, recent advances in transfer learning have shown substantial improvements from scale. We investigate whether modern methods can change the fortune of transfer learning for medical imaging. For this, we study the class of large-scale pre-trained networks presented by Kolesnikov et al. on three diverse imaging tasks: chest radiography, mammography, and dermatology. We study both transfer performance and critical properties for the deployment in the medical domain, including: out-of-distribution generalization, data-efficiency, sub-group fairness, and uncertainty estimation. Interestingly, we find that for some of these properties transfer from natural to medical images is indeed extremely effective, but only when performed at sufficient scale.

References in corpus (16)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• How transferable are features in deep neural networks?
• Language Models are Few-Shot Learners
• In Defense of the Triplet Loss for Person Re-Identification
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Similarity of Neural Network Representations Revisited
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Billion-scale semi-supervised learning for image classification
• What makes ImageNet good for transfer learning?
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting
• Joint Liver Lesion Segmentation and Classification via Transfer Learning

Cited by in corpus (8)

• Do Vision Transformers See Like Convolutional Neural Networks?
• Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions
• Revisiting the Calibration of Modern Neural Networks
• Exploring scalable medical image encoders beyond text supervision
• Exploring the Limits of Large Scale Pre-training
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• A new semi-supervised inductive transfer learning framework: Co-Transfer
• Meta-Learning to Improve Pre-Training