Reinventing 2D Convolutions for 3D Images

Abstract

There have been considerable debates over 2D and 3D representation learning on 3D medical images. 2D approaches could benefit from large-scale 2D pretraining, whereas they are generally weak in capturing large 3D contexts. 3D approaches are natively strong in 3D contexts, however few publicly available 3D medical dataset is large and diverse enough for universal 3D pretraining. Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D / 3D parts still exist. In this study, we bridge the gap between 2D and 3D convolutions by reinventing the 2D convolutions. We propose ACS (axial-coronal-sagittal) convolutions to perform natively 3D representation learning, while utilizing the pretrained weights on 2D datasets. In ACS convolutions, 2D convolution kernels are split by channel into three parts, and convoluted separately on the three views (axial, coronal and sagittal) of 3D representations. Theoretically, ANY 2D CNN (ResNet, DenseNet, or DeepLab) is able to be converted into a 3D ACS CNN, with pretrained weight of a same parameter size. Extensive experiments on several medical benchmarks (including classification, segmentation and detection tasks) validate the consistent superiority of the pretrained ACS CNNs, over the 2D / 3D CNN counterparts with / without pretraining. Even without pretraining, the ACS convolution can be used as a plug-and-play replacement of standard 3D convolution, with smaller model size and less computation.

arXiv commentIEEE Journal of Biomedical and Health Informatics (IEEE JBHI). Code is available at https://github.com/m3dv/ACSConv

References in corpus (34)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey on Deep Learning in Medical Image Analysis
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Fully Convolutional Networks for Semantic Segmentation
• VoxelMorph: A Learning Framework for Deformable Medical Image Registration
• Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
• The Liver Tumor Segmentation Benchmark (LiTS)
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• NiftyNet: a deep-learning platform for medical imaging
• fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
• Training Deep Nets with Sublinear Memory Cost
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
• Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learning
• DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation
• H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation from CT Volumes
• One Network to Segment Them All: A General, Lightweight System for Accurate 3D Medical Image Segmentation
• Volumetric Attention for 3D Medical Image Segmentation and Detection
• Collaborative Spatio-temporal Feature Learning for Video Action Recognition
• Ultrasound Image Representation Learning by Modeling Sonographer Visual Attention
• Parallel Separable 3D Convolution for Video and Volumetric Data Understanding
• Probabilistic Radiomics: Ambiguous Diagnosis with Controllable Shape Analysis
• Iterative Multi-domain Regularized Deep Learning for Anatomical Structure Detection and Segmentation from Ultrasound Images
• Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Database
• 3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes
• Evaluating and Boosting Uncertainty Quantification in Classification

Cited by in corpus (13)

• MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
• Cross-dimensional transfer learning in medical image segmentation with deep learning
• Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging
• Asymmetric 3D Context Fusion for Universal Lesion Detection
• SKA Science Data Challenge 2: analysis and results
• Conquering Data Variations in Resolution: A Slice-Aware Multi-Branch Decoder Network
• View-Disentangled Transformer for Brain Lesion Detection
• Hierarchical Classification of Pulmonary Lesions: A Large-Scale Radio-Pathomics Study
• Fibro-CoSANet: Pulmonary Fibrosis Prognosis Prediction using a Convolutional Self Attention Network
• Revisiting Hidden Representations in Transfer Learning for Medical Imaging
• Deep Structure Learning using Feature Extraction in Trained Projection Space
• ILPO-NET: Network for the invariant recognition of arbitrary volumetric patterns in 3D
• Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices