Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers

Abstract

Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundation for the online training on accelerators. However, existing schemes leaving batch normalization (BN) untouched during training are mostly incomplete quantization that still adopts high precision FP in some parts of the data paths. Currently, there is no solution that can use only low bit-width INT data during the whole training process of large-scale DNNs with acceptable accuracy. In this work, through decomposing all the computation steps in DNNs and fusing three special quantization functions to satisfy the different precision requirements, we propose a unified complete quantization framework termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W (Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN. Moreover, the Momentum optimizer is also quantized to realize a completely quantized framework. Experiments on ResNet18/34/50 models demonstrate that WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first time, the study of quantization in large-scale DNNs is advanced to the full 8-bit INT level. In this way, all the operations in the training and inference can be bit-wise operations, pushing towards faster processing speed, decreased memory cost, and higher energy efficiency. Our throughout quantization framework has great potential for future efficient portable devices with online learning ability.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Mixed Precision Training
• Training and Inference with Integers in Deep Neural Networks
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Scalable Methods for 8-bit Training of Neural Networks
• Resiliency of Deep Neural Networks under Quantization
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Effective Quantization Methods for Recurrent Neural Networks
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

Cited by in corpus (4)

• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Towards Unified INT8 Training for Convolutional Neural Network
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training