SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

Abstract

Going deeper and wider in neural architectures improves the accuracy, while the limited GPU DRAM places an undesired restriction on the network design domain. Deep Learning (DL) practitioners either need change to less desired network architectures, or nontrivially dissect a network across multiGPUs. These distract DL practitioners from concentrating on their original machine learning tasks. We present SuperNeurons: a dynamic GPU memory scheduling runtime to enable the network training far beyond the GPU DRAM capacity. SuperNeurons features 3 memory optimizations, \textit{Liveness Analysis}, \textit{Unified Tensor Pool}, and \textit{Cost-Aware Recomputation}, all together they effectively reduce the network-wide peak memory usage down to the maximal memory usage among layers. We also address the performance issues in those memory saving techniques. Given the limited GPU DRAM, SuperNeurons not only provisions the necessary memory for the training, but also dynamically allocates the memory for convolution workspaces to achieve the high performance. Evaluations against Caffe, Torch, MXNet and TensorFlow have demonstrated that SuperNeurons trains at least 3.2432 deeper network than current ones with the leading performance. Particularly, SuperNeurons can train ResNet2500 that has $10^4$ basic network layers on a 12GB K40c.

PPoPP '2018: 23nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Training Deep Nets with Sublinear Memory Cost
• Accelerating Deep Neural Network Training with Inconsistent Stochastic Gradient Descent

Cited by in corpus (11)

• DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression
• Understanding Performance Problems in Deep Learning Systems
• Multi-scale Cloud Detection in Remote Sensing Images using a Dual Convolutional Neural Network
• Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers
• ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning
• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments
• Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
• Partitioned Neural Network Training via Synthetic Intermediate Labels