Sampling-Free Learning of Bayesian Quantized Neural Networks

Abstract

Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs, but also reduces the variance in gradients. We evaluate BQNs on MNIST, Fashion-MNIST, KMNIST and CIFAR10 image classification datasets, compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).

References in corpus (6)

• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Weight Uncertainty in Neural Networks
• Trained Ternary Quantization
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks