Sampling-Free Learning of Bayesian Quantized Neural Networks
arXiv:1912.02992
Abstract
Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs, but also reduces the variance in gradients. We evaluate BQNs on MNIST, Fashion-MNIST, KMNIST and CIFAR10 image classification datasets, compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Weight Uncertainty in Neural Networks
- Trained Ternary Quantization
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
- Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
- Bitwise Neural Networks
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
- Probabilistic Binary Neural Networks
- Natural-Parameter Networks: A Class of Probabilistic Neural Networks
- Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
- A Survey on Bayesian Deep Learning