SiMaN: Sign-to-Magnitude Network Binarization

Abstract

Binary neural networks (BNNs) have attracted broad research interest due to their efficient storage and computational ability. Nevertheless, a significant challenge of BNNs lies in handling discrete constraints while ensuring bit entropy maximization, which typically makes their weight optimization very difficult. Existing methods relax the learning using the sign function, which simply encodes positive weights into +1s, and -1s otherwise. Alternatively, we formulate an angle alignment objective to constrain the weight binarization to {0,+1} to solve the challenge. In this paper, we show that our weight binarization provides an analytical solution by encoding high-magnitude weights into +1s, and 0s otherwise. Therefore, a high-quality discrete solution is established in a computationally efficient manner without the sign function. We prove that the learned weights of binarized networks roughly follow a Laplacian distribution that does not allow entropy maximization, and further demonstrate that it can be effectively solved by simply removing the ${\ell }_2$ regularization during network training. Our method, dubbed sign-to-magnitude network binarization (SiMaN), is evaluated on CIFAR-10 and ImageNet, demonstrating its superiority over the sign-based state-of-the-arts. Our source code, experimental settings, training logs and binary models are available at https://github.com/lmbxmu/SiMaN.

Accepted by IEEE TPAMI, 2022

References in corpus (10)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Binary Neural Networks: A Survey
• XNOR-Net++: Improved Binary Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• Training Binary Neural Networks through Learning with Noisy Supervision
• Channel Pruning via Automatic Structure Search
• The High-Dimensional Geometry of Binary Neural Networks

Cited by in corpus (1)

• A comprehensive review of Binary Neural Network