Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point

Abstract

We propose a cluster-based quantization method to convert pre-trained full precision weights into ternary weights with minimal impact on the accuracy. In addition, we also constrain the activations to 8-bits thus enabling sub 8-bit full integer inference pipeline. Our method uses smaller clusters of N filters with a common scaling factor to minimize the quantization loss, while also maximizing the number of ternary operations. We show that with a cluster size of N=4 on Resnet-101, can achieve 71.8% TOP-1 accuracy, within 6% of the best full precision results while replacing ~85% of all multiplications with 8-bit accumulations. Using the same method with 4-bit weights achieves 76.3% TOP-1 accuracy which within 2% of the full precision result. We also study the impact of the size of the cluster on both performance and accuracy, larger cluster sizes N=64 can replace ~98% of the multiplications with ternary operations but introduces significant drop in accuracy which necessitates fine tuning the parameters with retraining the network at lower precision. To address this we have also trained low-precision Resnet-50 with 8-bit activations and ternary weights by pre-initializing the network with full precision weights and achieve 68.9% TOP-1 accuracy within 4 additional epochs. Our final quantized model can run on a full 8-bit compute pipeline, with a potential 16x improvement in performance compared to baseline full-precision models.

References in corpus (7)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Trained Ternary Quantization
• Convolutional Neural Networks using Logarithmic Data Representation
• ImageNet pre-trained models with batch normalization
• Accelerating Deep Convolutional Networks using low-precision and sparsity

Cited by in corpus (9)

• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Optimizing Speech Recognition For The Edge
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• tuGEMM: Area-Power-Efficient Temporal Unary GEMM Architecture for Low-Precision Edge AI
• Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Design
• BitNet: Bit-Regularized Deep Neural Networks
• Smaller Models, Better Generalization
• NGEMM: Optimizing GEMM for Deep Learning via Compiler-based Techniques