Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance

Abstract

In this paper, we study 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While efficient, the lacking of representational capability and the training difficulty impede 1-bit CNNs from performing as well as real-valued networks. We propose Bi-Real net with a novel training algorithm to tackle these two challenges. To enhance the representational capability, we propagate the real-valued activations generated by each 1-bit convolution via a parameter-free shortcut. To address the training difficulty, we propose a training algorithm using a tighter approximation to the derivative of the sign function, a magnitude-aware gradient for weight updating, a better initialization method, and a two-step scheme for training a deep network. Experiments on ImageNet show that an 18-layer Bi-Real net with the proposed training algorithm achieves 56.4% top-1 classification accuracy, which is 10% higher than the state-of-the-arts (e.g., XNOR-Net) with greater memory saving and lower computational cost. Bi-Real net is also the first to scale up 1-bit CNNs to an ultra-deep network with 152 layers, and achieves 64.5% top-1 accuracy on ImageNet. A 50-layer Bi-Real net shows comparable performance to a real-valued network on the depth estimation task with only a 0.3% accuracy gap.

arXiv commentTo appear in IJCV. It is the journal extension of our ECCV paper: arXiv:1808.00278. Codes are available on: https://github.com/liuzechun/Bi-Real-net

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Binarized Neural Networks
• Loss-aware Binarization of Deep Networks
• Label Refinery: Improving ImageNet Classification through Label Progression
• Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
• End-to-end Active Object Tracking via Reinforcement Learning
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN