Models Genesis

Abstract

Transfer learning from natural images to medical images has been established as one of the most practical paradigms in deep learning for medical image analysis. To fit this paradigm, however, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information, thereby inevitably compromising its performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learnt by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch and existing pre-trained 3D models in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D/2.5D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated and recurrent anatomy in medical images can serve as strong yet free supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all codes and pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

arXiv commentJournal version of arXiv:1908.06912, accepted by Medical Image Analysis

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Survey on Deep Learning in Medical Image Analysis
• A Simple Framework for Contrastive Learning of Visual Representations
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
• The Liver Tumor Segmentation Benchmark (LiTS)
• Domain-adversarial neural networks to address the appearance variability of histopathology images
• Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation
• NiftyNet: a deep-learning platform for medical imaging
• Med3D: Transfer Learning for 3D Medical Image Analysis
• A New 2.5D Representation for Lymph Node Detection using Random Sets of Deep Convolutional Neural Network Observations
• Fractional Max-Pooling
• Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18F-FDG PET/CT images
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Improvements to context based self-supervised learning
• 3D Self-Supervised Methods for Medical Imaging
• A Kernel Theory of Modern Data Augmentation
• PatchShuffle Regularization
• On the Generalization Effects of Linear Transformations in Data Augmentation

Cited by in corpus (23)

• The Liver Tumor Segmentation Benchmark (LiTS)
• Recent advances and clinical applications of deep learning in medical image analysis
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• CaraNet: Context Axial Reverse Attention Network for Segmentation of Small Medical Objects
• Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports
• LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation
• DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation
• Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
• Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
• MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation
• Self-supervised 3D anatomy segmentation using self-distilled masked image transformer (SMIT)
• MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation
• MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentation
• Self-Supervised Pre-Training for Deep Image Prior-Based Robust PET Image Denoising
• 3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs
• Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation
• Learning Anatomically Consistent Embedding for Chest Radiography
• The Missing Piece: A Case for Pre-Training in 3D Medical Object Detection
• Detection of Emerging Infectious Diseases in Lung CT based on Spatial Anomaly Patterns
• Efficient 3D Brain Tumor Segmentation with Axial-Coronal-Sagittal Embedding