Training Binary Neural Networks with Real-to-Binary Convolutions

Abstract

This paper shows how to train binary networks to within a few percent points ($\sim 3-5\%$) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary.

ICLR 2020

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Trained Ternary Quantization
• XNOR-Net++: Improved Binary Neural Networks
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• Improved training of binary networks for human pose estimation and image recognition
• Accurate and Compact Convolutional Neural Networks with Trained Binarization

Cited by in corpus (20)

• Binary Neural Networks: A Survey
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• Self-Distribution Binary Neural Networks
• BoolNet: Minimizing The Energy Consumption of Binary Neural Networks
• Mirror Descent View for Neural Network Quantization
• BiPointNet: Binary Neural Network for Point Clouds
• Towards Practical Lipreading with Distilled and Efficient Models
• Generative Zero-shot Network Quantization
• High-Capacity Expert Binary Networks
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• SiMaN: Sign-to-Magnitude Network Binarization
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks
• Gradient-Based Interpretability Methods and Binarized Neural Networks
• Sparsity-Control Ternary Weight Networks
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond
• Demystifying and Generalizing BinaryConnect
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations