Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
arXiv:2012.00938
Abstract
Binarization of neural network models is considered as one of the promising methods to deploy deep neural network models on resource-constrained environments such as mobile devices. However, Binary Neural Networks (BNNs) tend to suffer from severe accuracy degradation compared to the full-precision counterpart model. Several techniques were proposed to improve the accuracy of BNNs. One of the approaches is to balance the distribution of binary activations so that the amount of information in the binary activations becomes maximum. Based on extensive analysis, in stark contrast to previous work, we argue that unbalanced activation distribution can actually improve the accuracy of BNNs. We also show that adjusting the threshold values of binary activation functions results in the unbalanced distribution of the binary activation, which increases the accuracy of BNN models. Experimental results show that the accuracy of previous BNN models (e.g. XNOR-Net and Bi-Real-Net) can be improved by simply shifting the threshold values of binary activation functions without requiring any other modification.
arXiv commentCVPR 2021, 10 pages, 10 figures
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Binarized Neural Networks
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Learning Structured Sparsity in Deep Neural Networks
- Improved training of binary networks for human pose estimation and image recognition
- BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations