GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
arXiv:2102.09298
Abstract
Quantized neural networks (QNNs) are among the main approaches for deploying deep neural networks on low resource edge devices. Training QNNs using different levels of precision throughout the network (dynamic quantization) typically achieves superior trade-offs between performance and computational load. However, optimizing the different precision levels of QNNs can be complicated, as the values of the bit allocations are discrete and difficult to differentiate for. Also, adequately accounting for the dependencies between the bit allocation of different layers is not straight-forward. To meet these challenges, in this work we propose GradFreeBits: a novel joint optimization scheme for training dynamic QNNs, which alternates between gradient-based optimization for the weights, and gradient-free optimization for the bit allocation. Our method achieves better or on par performance with current state of the art low precision neural networks on CIFAR10/100 and ImageNet classification. Furthermore, our approach can be extended to a variety of other applications involving neural networks used in conjunction with parameters which are difficult to optimize for.
References in corpus (15)
- mixup: Beyond Empirical Risk Minimization
- Knowledge Distillation: A Survey
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- CMA-ES for Hyperparameter Optimization of Deep Neural Networks
- A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
- A Hybrid Cooperative Co-evolution Algorithm Framework for Optimising Power Take Off and Placements of Wave Energy Converters
- HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
- Mixed Precision DNNs: All you need is a good parametrization
- Efficient Bitwidth Search for Practical Mixed Precision Neural Network
- Weight Normalization based Quantization for Deep Neural Network Compression