Accelerated CNN Training Through Gradient Approximation

Abstract

Training deep convolutional neural networks such as VGG and ResNet by gradient descent is an expensive exercise requiring specialized hardware such as GPUs. Recent works have examined the possibility of approximating the gradient computation while maintaining the same convergence properties. While promising, the approximations only work on relatively small datasets such as MNIST. They also fail to achieve real wall-clock speedups due to lack of efficient GPU implementations of the proposed approximation methods. In this work, we explore three alternative methods to approximate gradients, with an efficient GPU kernel implementation for one of them. We achieve wall-clock speedup with ResNet-20 and VGG-19 on the CIFAR-10 dataset upwards of 7%, with a minimal loss in validation accuracy.

arXiv commentAn abridged version was presented at EMC^2 : Workshop On Energy Efficient Machine Learning And Cognitive Computing For Embedded Applications at ISCA 2019

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• cuDNN: Efficient Primitives for Deep Learning
• Mixed Precision Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Submanifold Sparse Convolutional Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Efficient Sparse-Winograd Convolutional Neural Networks
• Biologically-plausible learning algorithms can scale to large datasets
• Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
• Minimal Effort Back Propagation for Convolutional Neural Networks
• Efficient Convolutional Neural Network Training with Direct Feedback Alignment
• Full deep neural network training on a pruned weight budget
• Faster Neural Network Training with Approximate Tensor Operations
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training