Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

Abstract

The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suitable precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.

Published as a conference paper in ICLR 2019

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• On the difficulty of training Recurrent Neural Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Binarized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

Cited by in corpus (4)

• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks