Benchmarking the Neural Linear Model for Regression

Abstract

The neural linear model is a simple adaptive Bayesian linear regression method that has recently been used in a number of problems ranging from Bayesian optimization to reinforcement learning. Despite its apparent successes in these settings, to the best of our knowledge there has been no systematic exploration of its capabilities on simple regression tasks. In this work we characterize these on the UCI datasets, a popular benchmark for Bayesian regression models, as well as on the recently introduced UCI "gap" datasets, which are better tests of out-of-distribution uncertainty. We demonstrate that the neural linear model is a simple method that shows generally good performance on these tasks, but at the cost of requiring good hyperparameter tuning.

Advances in Approximate Bayesian Inference (AABI 2019)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Scalable Bayesian Optimization Using Deep Neural Networks
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Bayesian Batch Active Learning as Sparse Subset Approximation
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Adaptive Bayesian Linear Regression for Automated Machine Learning

Cited by in corpus (5)

• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Bayesian Deep Learning via Subnetwork Inference
• NOMU: Neural Optimization-based Model Uncertainty
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Understanding Uncertainty in Bayesian Deep Learning