5.7k citations · 8.4k across the 42 of their papers we have counts for
21 papers · 1 filter
Bayesian Optimization via Continual Variational Last Layer Training
Paul Brunzema, Mikkel Jordahn, John Willes +3
Gaussian Processes (GPs) are widely seen as the state-of-the-art surrogate models for Bayesian optimization (BO) due to their ability to model uncertainty and their performance on…
Variational Bayesian Last Layers
James Harrison, John Willes, Jasper Snoek
We introduce a deterministic variational formulation for training Bayesian last layer neural networks. This yields a sampling-free, single-pass model and loss that effectively impr…
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
Avi Singh, John D. Co-Reyes, Rishabh Agarwal +38
Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of hi…
Plex: Towards Reliability using Pretrained Large Model Extensions
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry +23
A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures. P…
Pre-training helps Bayesian optimization too
Zi Wang, George E. Dahl, Kevin Swersky +6
Bayesian optimization (BO) has become a popular strategy for global optimization of many expensive real-world functions. Contrary to a common belief that BO is suited to optimizing…
A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren +7
Accurate uncertainty quantification is a major challenge in deep learning, as neural networks can make overconfident errors and assign high confidence predictions to out-of-distrib…
Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach
Setareh Ariafar, Justin Gilmer, Zachary Nado +3
Black box optimization requires specifying a search space to explore for solutions, e.g. a d-dimensional compact space, and this choice is critical for getting the best results at…
Sparse MoEs meet Efficient Ensembles
James Urquhart Allingham, Florian Wenzel, Zelda E Mariet +10
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models.…
Pre-trained Gaussian Processes for Bayesian Optimization
Zi Wang, George E. Dahl, Kevin Swersky +5
Bayesian optimization (BO) has become a popular strategy for global optimization of expensive real-world functions. Contrary to a common expectation that BO is suited to optimizing…
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning
Zachary Nado, Neil Band, Mark Collier +23
High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ab…
Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure
Samuel Kim, Peter Y. Lu, Charlotte Loh +3
Bayesian optimization (BO) is a popular paradigm for global optimization of expensive black-box functions, but there are many domains where the function is not completely a black-b…
Combining Ensembles and Data Augmentation can Harm your Calibration
Yeming Wen, Ghassen Jerfel, Rafael Muller +4
Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness. Similarly, data augmentation techniqu…
Training independent subnetworks for robust prediction
Marton Havasi, Rodolphe Jenatton, Stanislav Fort +5
Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over th…
Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Benjamin Kompa, Jasper Snoek, Andrew Beam
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the qualit…
Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
Shreyas Padhy, Zachary Nado, Jie Ren +3
Accurate estimation of predictive uncertainty in modern neural networks is critical to achieve well calibrated predictions and detect out-of-distribution (OOD) inputs. The most pro…
Hyperparameter Ensembles for Robustness and Uncertainty Quantification
Florian Wenzel, Jasper Snoek, Dustin Tran +1
Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introdu…
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
Zachary Nado, Shreyas Padhy, D. Sculley +3
Covariate shift has been shown to sharply degrade both predictive accuracy and the calibration of uncertainty estimates for deep learning models. This is worrying, because covariat…
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen +5
Bayesian neural networks (BNNs) demonstrate promising success in improving the robustness and uncertainty quantification of modern deep learning. However, they generally struggle w…
Weighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
Setareh Ariafar, Zelda Mariet, Ehsan Elhamifar +3
Many contemporary machine learning models require extensive tuning of hyperparameters to perform well. A variety of methods, such as Bayesian optimization, have been developed to a…
The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Jakub Swiatkowski, Kevin Roth, Bastiaan S. Veeling +7
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods…
Hydra: Preserving Ensemble Diversity for Model Distillation
Linh Tran, Bastiaan S. Veeling, Kevin Roth +7
Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory…