XNOR-Net++: Improved Binary Neural Networks

Abstract

This paper proposes an improved training algorithm for binary neural networks in which both weights and activations are binary numbers. A key but fairly overlooked feature of the current state-of-the-art method of XNOR-Net is the use of analytically calculated real-valued scaling factors for re-weighting the output of binary convolutions. We argue that analytic calculation of these factors is sub-optimal. Instead, in this work, we make the following contributions: (a) we propose to fuse the activation and weight scaling factors into a single one that is learned discriminatively via backpropagation. (b) More importantly, we explore several ways of constructing the shape of the scale factors while keeping the computational budget fixed. (c) We empirically measure the accuracy of our approximations and show that they are significantly more accurate than the analytically calculated one. (d) We show that our approach significantly outperforms XNOR-Net within the same computational budget when tested on the challenging task of ImageNet classification, offering up to 6\% accuracy gain.

Accepted to BMVC 2019

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• YOLOv3: An Incremental Improvement
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Densely Connected Convolutional Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Aggregated Residual Transformations for Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Training deep neural networks with low precision multiplications
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Conditional Computation in Neural Networks for faster models
• Hierarchical binary CNNs for landmark localization with limited resources
• Training Competitive Binary Neural Networks from Scratch
• WRPN: Training and Inference using Wide Reduced-Precision Networks

Cited by in corpus (29)

• Binary Neural Networks: A Survey
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Rotated Binary Neural Network
• HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation
• Lightweight Pixel Difference Networks for Efficient Visual Representation Learning
• Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• Self-Distribution Binary Neural Networks
• Knowledge distillation via adaptive instance normalization
• BiPointNet: Binary Neural Network for Point Clouds
• Learning Frequency Domain Approximation for Binary Neural Networks
• Convolutional Neural Networks Quantization with Attention
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Binarizing by Classification: Is soft function really necessary?
• A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network
• Learning Architectures for Binary Networks
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• High-Capacity Expert Binary Networks
• RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• Quantum Annealing Formulation for Binary Neural Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer