Training Deep Neural Network in Limited Precision

Abstract

Energy and resource efficient training of DNNs will greatly extend the applications of deep learning. However, there are three major obstacles which mandate accurate calculation in high precision. In this paper, we tackle two of them related to the loss of gradients during parameter update and backpropagation through a softmax nonlinearity layer in low precision training. We implemented SGD with Kahan summation by employing an additional parameter to virtually extend the bit-width of the parameters for a reliable parameter update. We also proposed a simple guideline to help select the appropriate bit-width for the last FC layer followed by a softmax nonlinearity layer. It determines the lower bound of the required bit-width based on the class size of the dataset. Extensive experiments on various network architectures and benchmarks verifies the effectiveness of the proposed technique for low precision training.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Deep Learning with Limited Numerical Precision
• Mixed Precision Training
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations