ReCU: Reviving the Dead Weights in Binary Neural Networks

Abstract

Binary neural networks (BNNs) have received increasing attention due to their superior reductions of computation and memory. Most existing works focus on either lessening the quantization error by minimizing the gap between the full-precision weights and their binarization or designing a gradient approximation to mitigate the gradient mismatch, while leaving the "dead weights" untouched. This leads to slow convergence when training BNNs. In this paper, for the first time, we explore the influence of "dead weights" which refer to a group of weights that are barely updated during the training of BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead weights" for updating. We prove that reviving the "dead weights" by ReCU can result in a smaller quantization error. Besides, we also take into account the information entropy of the weights, and then mathematically analyze why the weight standardization can benefit BNNs. We demonstrate the inherent contradiction between minimizing the quantization error and maximizing the information entropy, and then propose an adaptive exponential scheduler to identify the range of the "dead weights". By considering the "dead weights", our method offers not only faster BNN training, but also state-of-the-art performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be available at https://github.com/z-hXu/ReCU.

arXiv commentAccepted by International Conference on Computer Vision 2021 (ICCV 2021)

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• XNOR-Net++: Improved Binary Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• Improved training of binary networks for human pose estimation and image recognition
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
• Mirror Descent View for Neural Network Quantization
• Understanding Learning Dynamics of Binary Neural Networks via Information Bottleneck