Backprop with Approximate Activations for Memory-efficient Network Training

Abstract

Training convolutional neural network models is memory intensive since back-propagation requires storing activations of all intermediate layers. This presents a practical concern when seeking to deploy very deep architectures in production, especially when models need to be frequently re-trained on updated datasets. In this paper, we propose a new implementation for back-propagation that significantly reduces memory usage, by enabling the use of approximations with negligible computational cost and minimal effect on training performance. The algorithm reuses common buffers to temporarily store full activations and compute the forward pass exactly. It also stores approximate per-layer copies of activations, at significant memory savings, that are used in the backward pass. Compared to simply approximating activations within standard back-propagation, our method limits accumulation of errors across layers. This allows the use of much lower-precision approximations without affecting training accuracy. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method yields performance close to exact training, while storing activations compactly with as low as 4-bit precision.

arXiv commentProject page at http://projects.ayanc.org/blpa/

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• TensorFlow: A system for large-scale machine learning
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Mixed Precision Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Training Deep Nets with Sublinear Memory Cost
• Scalable Methods for 8-bit Training of Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations

Cited by in corpus (5)

• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• Memory Optimization for Deep Networks
• Faster Neural Network Training with Approximate Tensor Operations
• OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
• Enabling Binary Neural Network Training on the Edge