Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1

Abstract

The training of deep neural networks (DNNs) requires intensive resources both for computation and for storage performance. Thus, DNNs cannot be efficiently applied to mobile phones and embedded devices, which seriously limits their applicability in industry applications. To address this issue, we propose a novel encoding scheme of using {-1,+1} to decompose quantized neural networks (QNNs) into multi-branch binary networks, which can be efficiently implemented by bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving. Based on our method, users can easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources. The proposed mechanism is very suitable for the use of FPGA and ASIC in terms of data storage and computation, which provides a feasible idea for smart chips. We validate the effectiveness of our method on both large-scale image classification tasks (e.g., ImageNet) and object detection tasks. In particular, our method with low-bit encoding can still achieve almost the same performance as its full-precision counterparts.

arXiv comment9 pages, 2 figures, Proc. 33rd AAAI Conf. Artif. Intell., 2019

References in corpus (19)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• Learning both Weights and Connections for Efficient Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Towards Accurate Binary Convolutional Neural Network
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Model compression via distillation and quantization
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Training Quantized Nets: A Deeper Understanding
• Network Sketching: Exploiting Binary Structure in Deep CNNs