XNOR-Net++: Improved Binary Neural Networks
arXiv:1909.13863
Abstract
This paper proposes an improved training algorithm for binary neural networks in which both weights and activations are binary numbers. A key but fairly overlooked feature of the current state-of-the-art method of XNOR-Net is the use of analytically calculated real-valued scaling factors for re-weighting the output of binary convolutions. We argue that analytic calculation of these factors is sub-optimal. Instead, in this work, we make the following contributions: (a) we propose to fuse the activation and weight scaling factors into a single one that is learned discriminatively via backpropagation. (b) More importantly, we explore several ways of constructing the shape of the scale factors while keeping the computational budget fixed. (c) We empirically measure the accuracy of our approximations and show that they are significantly more accurate than the analytically calculated one. (d) We show that our approach significantly outperforms XNOR-Net within the same computational budget when tested on the challenging task of ImageNet classification, offering up to 6\% accuracy gain.
Accepted to BMVC 2019
References in corpus (2)
Cited by in corpus (9)
- Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices
- Convolutional Neural Networks Quantization with Attention
- WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
- RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
- Quantum Annealing Formulation for Binary Neural Networks
- "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
- Convolutional neural networks compression with low rank and sparse tensor decompositions
- Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
- Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations