Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
arXiv:1911.12990
Abstract
Network quantization, which aims to reduce the bit-lengths of the network weights and activations, has emerged as one of the key ingredients to reduce the size of neural networks for their deployments to resource-limited devices. In order to overcome the nature of transforming continuous activations and weights to discrete ones, recent study called Relaxed Quantization (RQ) [Louizos et al. 2019] successfully employ the popular Gumbel-Softmax that allows this transformation with efficient gradient-based optimization. However, RQ with this Gumbel-Softmax relaxation still suffers from bias-variance trade-off depending on the temperature parameter of Gumbel-Softmax. To resolve the issue, we propose a novel method, Semi-Relaxed Quantization (SRQ) that uses multi-class straight-through estimator to effectively reduce the bias and variance, along with a new regularization technique, DropBits that replaces dropout regularization to randomly drop the bits instead of neurons to further reduce the bias of the multi-class straight-through estimator in SRQ. As a natural extension of DropBits, we further introduce the way of learning heterogeneous quantization levels to find proper bit-length for each layer using DropBits. We experimentally validate our method on various benchmark datasets and network architectures, and also support the quantized lottery ticket hypothesis: learning heterogeneous quantization levels outperforms the case using the same but fixed quantization levels from scratch.
arXiv commentNew submission with another link
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Decoupled Weight Decay Regularization
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Ternary Weight Networks
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Learning Structured Sparsity in Deep Neural Networks
- Adding Gradient Noise Improves Learning for Very Deep Networks
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
- Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
- Mixed Precision DNNs: All you need is a good parametrization
- Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
- AutoQ: Automated Kernel-Wise Neural Network Quantization
- Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks
- Differentiable Fine-grained Quantization for Deep Neural Network Compression