Training Binary Neural Networks using the Bayesian Learning Rule

Abstract

Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as the Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which justify such methods? In this paper, we propose such an approach using the Bayesian learning rule. The rule, when applied to estimate a Bernoulli distribution over the binary weights, results in an algorithm which justifies some of the algorithmic choices made by the previous approaches. The algorithm not only obtains state-of-the-art performance, but also enables uncertainty estimation for continual learning to avoid catastrophic forgetting. Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.

arXiv commentaccepted by ICML 2020, the camera-ready version

References in corpus (14)

• Categorical Reparameterization with Gumbel-Softmax
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Spatially-sparse convolutional neural networks
• Practical Deep Learning with Bayesian Principles
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• Probabilistic Binary Neural Networks
• Mirror Descent View for Neural Network Quantization

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Supplementing Recurrent Neural Networks with Annealing to Solve Combinatorial Optimization Problems
• The Bayesian Learning Rule
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• BiSNN: Training Spiking Neural Networks with Binary Weights via Bayesian Learning
• Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule