Faster Neural Network Training with Approximate Tensor Operations
arXiv:1805.08079
Abstract
We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.
arXiv commentNeurIPS 2021 camera ready
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Recurrent Neural Network Regularization
- Wide Residual Networks
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- cuDNN: Efficient Primitives for Deep Learning
- Compressing Deep Convolutional Networks using Vector Quantization
- Mixed Precision Training
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Dropout Training as Adaptive Regularization
- Dynamic Channel Pruning: Feature Boosting and Suppression
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- Channel Gating Neural Networks
- meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
- Factorization tricks for LSTM networks
- Analysis and Optimization of Convolutional Neural Network Architectures
- SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems
- Learning Compact Recurrent Neural Networks
- Backprop with Approximate Activations for Memory-efficient Network Training
- Efficient batchwise dropout training using submatrices