LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentation
arXiv:2006.12575
Abstract
Deep Learning (DL) models are becoming larger, because the increase in model size might offer significant accuracy gain. To enable the training of large deep networks, data parallelism and model parallelism are two well-known approaches for parallel training. However, data parallelism does not help reduce memory footprint per device. In this work, we introduce Large deep 3D ConvNets with Automated Model Parallelism (LAMP) and investigate the impact of both input's and deep 3D ConvNets' size on segmentation accuracy. Through automated model parallelism, it is feasible to train large deep 3D ConvNets with a large input patch, even the whole image. Extensive experiments demonstrate that, facilitated by the automated model parallelism, the segmentation accuracy can be improved through increasing model size and input context size, and large input yields significant inference speedup compared with sliding window of small patches in the inference. Code is available\footnote{https://monai.io/research/lamp-automated-model-parallelism}.
MICCAI 2020 Early Accepted paper. Code is available\footnote{https://monai.io/research/lamp-automated-model-parallelism}
References in corpus (10)
- Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
- Mixed Precision Training
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms
- AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy
- Training Deep Nets with Sublinear Memory Cost
- nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Invertible Network for Classification and Biomarker Selection for ASD
- Neural Multi-Scale Self-Supervised Registration for Echocardiogram Dense Tracking