Matrix and tensor decompositions for training binary neural networks

Abstract

This paper is on improving the training of binary neural networks in which both activations and weights are binary. While prior methods for neural network binarization binarize each filter independently, we propose to instead parametrize the weight tensor of each layer using matrix or tensor decomposition. The binarization process is then performed using this latent parametrization, via a quantization function (e.g. sign function) applied to the reconstructed weights. A key feature of our method is that while the reconstruction is binarized, the computation in the latent factorized space is done in the real domain. This has several advantages: (i) the latent factorization enforces a coupling of the filters before binarization, which significantly improves the accuracy of the trained models. (ii) while at training time, the binary weights of each convolutional layer are parametrized using real-valued matrix or tensor decomposition, during inference we simply use the reconstructed (binary) weights. As a result, our method does not sacrifice any advantage of binary networks in terms of model compression and speeding-up inference. As a further contribution, instead of computing the binary weight scaling factors analytically, as in prior work, we propose to learn them discriminatively via back-propagation. Finally, we show that our approach significantly outperforms existing methods when tested on the challenging tasks of (a) human pose estimation (more than 4% improvements) and (b) ImageNet classification (up to 5% performance gains).

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Fixed Point Quantization of Deep Convolutional Networks
• Human pose estimation via Convolutional Part Heatmap Regression
• Stacked Hourglass Networks for Human Pose Estimation
• YOLO9000: Better, Faster, Stronger
• Towards Accurate Binary Convolutional Neural Network
• Aggregated Residual Transformations for Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Training deep neural networks with low precision multiplications
• Tensor Regression Networks
• Learning Feature Pyramids for Human Pose Estimation
• Multi-Scale Structure-Aware Network for Human Pose Estimation
• Tensor Contraction Layers for Parsimonious Deep Nets
• WRPN: Training and Inference using Wide Reduced-Precision Networks

Cited by in corpus (3)

• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Convolutional neural networks compression with low rank and sparse tensor decompositions