Ternary Neural Networks with Fine-Grained Quantization

Abstract

We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we demonstrate a minimal loss in classification accuracy on state-of-the-art topologies without additional training. We provide an improved theoretical formulation that forms the basis for a higher quality solution using FGQ. Our method involves ternarizing the original weight tensor in groups of $N$ weights. Using $N=4$, we achieve Top-1 accuracy within $3.7\%$ and $4.2\%$ of the baseline full precision result for Resnet-101 and Resnet-50 respectively, while eliminating $75\%$ of all multiplications. These results enable a full 8/4-bit inference pipeline, with best-reported accuracy using ternary weights on ImageNet dataset, with a potential of $9\times$ improvement in performance. Also, for smaller networks like AlexNet, FGQ achieves state-of-the-art results. We further study the impact of group size on both performance and accuracy. With a group size of $N=64$, we eliminate $\approx 99\%$ of the multiplications; however, this introduces a noticeable drop in accuracy, which necessitates fine tuning the parameters at lower precision. We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the full precision result with $<30\%$ additional training overhead. Our final quantized model can run on a full 8-bit compute pipeline using 2-bit weights and has the potential of up to $15\times$ improvement in performance compared to baseline full-precision models.

References in corpus (8)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Ternary Weight Networks
• Trained Ternary Quantization
• Convolutional Neural Networks using Logarithmic Data Representation
• ImageNet pre-trained models with batch normalization
• 8-Bit Approximations for Parallelism in Deep Learning

Cited by in corpus (25)

• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• A Survey on Methods and Theories of Quantized Neural Networks
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• A Study of BFLOAT16 for Deep Learning Training
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Towards Efficient Training for Neural Network Quantization
• AutoQ: Automated Kernel-Wise Neural Network Quantization
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Efficient Hybrid Network Architectures for Extremely Quantized Neural Networks Enabling Intelligence at the Edge
• End-to-End Learned Image Compression with Quantized Weights and Activations
• Dataflow-based Joint Quantization of Weights and Activations for Deep Neural Networks
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Neural Networks Weights Quantization: Target None-retraining Ternary (TNT)
• GDRQ: Group-based Distribution Reshaping for Quantization
• Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Augmented Memory Computing: Dynamically Augmented SRAM Storage for Data Intensive Applications
• Training of mixed-signal optical convolutional neural network with reduced quantization level
• Light Multi-segment Activation for Model Compression
• Robustness-aware 2-bit quantization with real-time performance for neural network
• Sparsity-Control Ternary Weight Networks
• CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
• Low Power In-Memory Implementation of Ternary Neural Networks with Resistive RAM-Based Synapse