On the Importance of Strong Baselines in Bayesian Deep Learning

Abstract

Like all sub-fields of machine learning Bayesian Deep Learning is driven by empirical validation of its theoretical proposals. Given the many aspects of an experiment it is always possible that minor or even major experimental flaws can slip by both authors and reviewers. One of the most popular experiments used to evaluate approximate inference techniques is the regression experiment on UCI datasets. However, in this experiment, models which have been trained to convergence have often been compared with baselines trained only for a fixed number of iterations. We find that a well-established baseline, Monte Carlo dropout, when evaluated under the same experimental settings shows significant improvements. In fact, the baseline outperforms or performs competitively with methods that claimed to be superior to the very same baseline method when they were introduced. Hence, by exposing this flaw in experimental procedure, we highlight the importance of using identical experimental setups to evaluate, compare, and benchmark methods in Bayesian Deep Learning.

arXiv commentBayesian Deep Learning Workshop, NeurIPS 2018

References in corpus (3)

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Model Selection in Bayesian Neural Networks via Horseshoe Priors

Cited by in corpus (15)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Using AntiPatterns to avoid MLOps Mistakes
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• On Batch Normalisation for Approximate Bayesian Inference
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Structured Dropout Variational Inference for Bayesian Neural Networks
• A Variational View on Bootstrap Ensembles as Bayesian Inference
• Stochastic Bayesian Neural Networks
• Revisiting the Effects of Stochasticity for Hamiltonian Samplers