Memory-Efficient Implementation of DenseNets

Abstract

The DenseNet architecture is highly computationally efficient as a result of feature reuse. However, a naive DenseNet implementation can require a significant amount of GPU memory: If not properly managed, pre-activation batch normalization and contiguous convolution operations can produce feature maps that grow quadratically with network depth. In this technical report, we introduce strategies to reduce the memory consumption of DenseNets during training. By strategically using shared memory allocations, we reduce the memory cost for storing feature maps from quadratic to linear. Without the GPU memory bottleneck, it is now possible to train extremely deep DenseNets. Networks with 14M parameters can be trained on a single GPU, up from 4M. A 264-layer DenseNet (73M parameters), which previously would have been infeasible to train, can now be trained on a single workstation with 8 NVIDIA Tesla M40 GPUs. On the ImageNet ILSVRC classification dataset, this large DenseNet obtains a state-of-the-art single-crop top-1 error of 20.26%.

Technical report

References in corpus (5)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• SGDR: Stochastic Gradient Descent with Warm Restarts
• cuDNN: Efficient Primitives for Deep Learning
• Training Deep Nets with Sublinear Memory Cost

Cited by in corpus (28)

• Densely Connected Convolutional Networks
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Pelee: A Real-Time Object Detection System on Mobile Devices
• Real-Time 2D-3D Deformable Registration with Deep Learning and Application to Lung Radiotherapy Targeting
• Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
• JPEG Steganalysis Based on DenseNet
• Efficient Memory Management for GPU-based Deep Learning Systems
• ResNet or DenseNet? Introducing Dense Shortcuts to ResNet
• A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation
• Cascaded Structure Tensor Framework for Robust Identification of Heavily Occluded Baggage Items from X-ray Scans
• Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
• Efficient 3D Fully Convolutional Networks for Pulmonary Lobe Segmentation in CT Images
• EdgeCNN: Convolutional Neural Network Classification Model with small inputs for Edge Computing
• Memory Optimization for Deep Networks
• SpecNet: Spectral Domain Convolutional Neural Network
• AOGNets: Compositional Grammatical Architectures for Deep Learning
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification
• Improving the Resolution of CNN Feature Maps Efficiently with Multisampling
• Learning Fully Dense Neural Networks for Image Semantic Segmentation
• Influence of Image Classification Accuracy on Saliency Map Estimation
• DDU-Nets: Distributed Dense Model for 3D MRI Brain Tumor Segmentation
• Analysis of memory consumption by neural networks based on hyperparameters
• Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Coding
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB
• Exploiting Nontrivial Connectivity for Automatic Speech Recognition
• Enhanced generative adversarial network for 3D brain MRI super-resolution
• A Survey on Green Deep Learning