AutoQ: Automated Kernel-Wise Neural Network Quantization

Abstract

Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06\%, and decrease the inference energy consumption by 50.69\%, while achieving the same inference accuracy.

arXiv comment10 pages, 12 figures

References in corpus (13)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Fixed Point Quantization of Deep Convolutional Networks
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Data-Efficient Hierarchical Reinforcement Learning
• Bitwise Neural Networks
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Ternary Neural Networks with Fine-Grained Quantization
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Optimizing Bit-Serial Matrix Multiplication for Reconfigurable Computing
• Bayesian Optimization for Parameter Tuning of the XOR Neural Network

Cited by in corpus (10)

• Towards Efficient Training for Neural Network Quantization
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• AutoPrivacy: Automated Layer-wise Parameter Selection for Secure Neural Network Inference
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization