Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks

Abstract

It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the well-accepted solutions facilitating the training of low precision fixed point networks is stochastic rounding. However, to the best of our knowledge, the source of the instability in training neural networks with noisy gradient updates has not been well investigated. This work is an attempt to draw a theoretical connection between low numerical precision and training algorithm stability. In doing so, we will also propose and verify through experiments methods that are able to improve the training performance of deep convolutional networks in fixed point.

ICML2016 - Workshop on On-Device Intelligence

References in corpus (1)

• Deep Learning with Limited Numerical Precision

Cited by in corpus (6)

• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms
• Understanding the Energy and Precision Requirements for Online Learning
• IFQ-Net: Integrated Fixed-point Quantization Networks for Embedded Vision
• A Computing Kernel for Network Binarization on PyTorch