Compromise-free Bayesian neural networks

Abstract

We conduct a thorough analysis of the relationship between the out-of-sample performance and the Bayesian evidence (marginal likelihood) of Bayesian neural networks (BNNs), as well as looking at the performance of ensembles of BNNs, both using the Boston housing dataset. Using the state-of-the-art in nested sampling, we numerically sample the full (non-Gaussian and multimodal) network posterior and obtain numerical estimates of the Bayesian evidence, considering network models with up to 156 trainable parameters. The networks have between zero and four hidden layers, either $\tanh$ or $ReLU$ activation functions, and with and without hierarchical priors. The ensembles of BNNs are obtained by determining the posterior distribution over networks, from the posterior samples of individual BNNs re-weighted by the associated Bayesian evidence values. There is good correlation between out-of-sample performance and evidence, as well as a remarkable symmetry between the evidence versus model size and out-of-sample performance versus model size planes. Networks with $ReLU$ activation functions have consistently higher evidences than those with $\tanh$ functions, and this is reflected in their out-of-sample performance. Ensembling over architectures acts to further improve performance relative to the individual BNNs.

https://github.com/PolyChord/PolyChordLite; https://github.com/SuperKam91/bnn

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• PolyChord: nested sampling for cosmology
• PolyChord: next-generation nested sampling
• Bayesian model selection without evidences: application to the dark energy equation-of-state
• nestcheck: diagnostic tests for nested sampling calculations
• Bayesian inflationary reconstructions from Planck 2018 data
• Label Switching Problem in Bayesian Analysis for Gravitational Wave Astronomy
• Towards a framework for testing general relativity with extreme-mass-ratio-inspiral observations
• Bayesian sparse reconstruction: a brute-force approach to astronomical imaging and machine learning

Cited by in corpus (4)

• Nested Sampling Methods
• GLOBALEMU: A novel and robust approach for emulating the sky-averaged 21-cm signal from the cosmic dawn and epoch of reionisation
• Invariant polynomials and machine learning
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges