Searching for Low-Bit Weights in Quantized Neural Networks

Abstract

Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty of quantized networks. Compared with full-precision parameters (i.e., 32-bit floating numbers), low-bit values are selected from a much smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected to establish the desired quantized network. Experimental results on benchmarks demonstrate that the proposed method is able to produce quantized neural networks with higher performance over the state-of-the-art methods on both image classification and super-resolution tasks.

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• DARTS: Differentiable Architecture Search
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• MoBiNet: A Mobile Binary Network for Image Classification
• Cogradient Descent for Bilinear Optimization

Cited by in corpus (8)

• GhostNets on Heterogeneous Devices via Cheap Operations
• SCOP: Scientific Control for Reliable Neural Network Pruning
• Vision Transformer Pruning
• Distilling Object Detectors via Decoupled Features
• Learning Frequency Domain Approximation for Binary Neural Networks
• Training Stronger Baselines for Learning to Optimize
• Manifold Regularized Dynamic Network Pruning
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks