Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks

Abstract

It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the well-accepted solutions facilitating the training of low precision fixed point networks is stochastic rounding. However, to the best of our knowledge, the source of the instability in training neural networks with noisy gradient updates has not been well investigated. This work is an attempt to draw a theoretical connection between low numerical precision and training algorithm stability. In doing so, we will also propose and verify through experiments methods that are able to improve the training performance of deep convolutional networks in fixed point.

arXiv commentICML2016 - Workshop on On-Device Intelligence

References in corpus (5)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Deep Learning with Limited Numerical Precision
• Fixed Point Quantization of Deep Convolutional Networks
• Hardware-oriented Approximation of Convolutional Neural Networks
• Neural Networks with Few Multiplications

Cited by in corpus (11)

• Fixed Point Quantization of Deep Convolutional Networks
• Bayesian Compression for Deep Learning
• A Survey on Methods and Theories of Quantized Neural Networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• Training Quantized Neural Networks to Global Optimality via Semidefinite Programming
• Understanding the Energy and Precision Requirements for Online Learning
• A Computing Kernel for Network Binarization on PyTorch
• IFQ-Net: Integrated Fixed-point Quantization Networks for Embedded Vision