Self-Distribution Binary Neural Networks

Abstract

In this work, we study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation). Feature representation is critical for deep neural networks, while in BNNs, the features only differ in signs. Prior work introduces scaling factors into binary weights and activations to reduce the quantization error and effectively improves the classification accuracy of BNNs. However, the scaling factors not only increase the computational complexity of networks, but also make no sense to the signs of binary features. To this end, Self-Distribution Binary Neural Network (SD-BNN) is proposed. Firstly, we utilize Activation Self Distribution (ASD) to adaptively adjust the sign distribution of activations, thereby improve the sign differences of the outputs of the convolution. Secondly, we adjust the sign distribution of weights through Weight Self Distribution (WSD) and then fine-tune the sign distribution of the outputs of the convolution. Extensive experiments on CIFAR-10 and ImageNet datasets with various network structures show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs (e.g., achieves 92.5% on CIFAR-10 and 66.5% on ImageNet with ResNet-18) with less computation cost. Code is available at https://github.com/ pingxue-hfut/SD-BNN.

arXiv comment10 pages, 6 figures. Appl Intell (2022)

References in corpus (11)

• Knowledge Distillation: A Survey
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Weakly Supervised Attention Pyramid Convolutional Neural Network for Fine-Grained Visual Classification
• XNOR-Net++: Improved Binary Neural Networks
• Depth Quality Aware Salient Object Detection
• Training Binary Neural Networks with Real-to-Binary Convolutions
• ResKD: Residual-Guided Knowledge Distillation
• Evidential fully convolutional network for semantic segmentation
• Improved training of binary networks for human pose estimation and image recognition
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Learning Versatile Convolution Filters for Efficient Visual Recognition

Cited by in corpus (2)

• Binarizing by Classification: Is soft function really necessary?
• How to train accurate BNNs for embedded systems?