228 citations · 1.4k across the 27 of their papers we have counts for
20 papers · 1 filter
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Vatsal Baherwani, Zixi Chen, Shikai Qiu +2
Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learn…
From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
Marc Finzi, Shikai Qiu, Yiding Jiang +3
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to exist…
Out-of-Distribution Detection Methods Answer the Wrong Questions
Yucen Lily Li, Daohan Lu, Polina Kirichenko +4
To detect distribution shifts and improve model safety, many out-of-distribution (OOD) detection methods rely on the predictive uncertainty or features of supervised models trained…
Simple and Fast Group Robustness by Automatic Feature Reweighting
Shikai Qiu, Andres Potapczynski, Pavel Izmailov +1
A major challenge to out-of-distribution generalization is reliance on spurious features -- patterns that are predictive of the class label in the training data distribution, but n…
On Feature Learning in the Presence of Spurious Correlations
Pavel Izmailov, Polina Kirichenko, Nate Gruver +1
Deep classifiers are known to rely on spurious features $\unicode{x2013}$ patterns which are correlated with the target on the training data but not inherently relevant to the lear…
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson
Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn…
On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification
Sanyam Kapoor, Wesley J. Maddox, Pavel Izmailov +1
Aleatoric uncertainty captures the inherent randomness of the data, such as measurement noise. In Bayesian regression, we often use a Gaussian observation model, where we control t…
Bayesian Model Selection, the Marginal Likelihood, and Generalization
Sanae Lotfi, Pavel Izmailov, Gregory Benton +2
How do we compare between hypotheses that are entirely consistent with observations? The marginal likelihood (aka Bayesian evidence), which represents the probability of generating…
Dangers of Bayesian Model Averaging under Covariate Shift
Pavel Izmailov, Patrick Nicholson, Sanae Lotfi +1
Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, B…
Does Knowledge Distillation Really Work?
Samuel Stanton, Pavel Izmailov, Polina Kirichenko +2
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge…
What Are Bayesian Neural Network Posteriors Really Like?
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman +1
The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using in…
Learning Invariances in Neural Networks
Gregory Benton, Marc Finzi, Pavel Izmailov +1
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present i…
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
Andrew Gordon Wilson, Pavel Izmailov
The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accu…
Semi-Supervised Learning with Normalizing Flows
Pavel Izmailov, Polina Kirichenko, Marc Finzi +1
Normalizing flows transform a latent distribution through an invertible neural network for a flexible and pleasingly simple approach to generative modelling, while preserving an ex…
Subspace Inference for Bayesian Deep Learning
Pavel Izmailov, Wesley J. Maddox, Polina Kirichenko +3
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Ba…
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Wesley Maddox, Timur Garipov, Pavel Izmailov +2
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA),…
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
Ben Athiwaratkun, Marc Finzi, Pavel Izmailov +1
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its…
Averaging Weights Leads to Wider Optima and Better Generalization
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov +2
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple a…
Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition
Pavel Izmailov, Alexander Novikov, Dmitry Kropotov
We propose a method (TT-GP) for approximate inference in Gaussian Process (GP) models. We build on previous scalable GP research including stochastic variational inference based on…
Faster variational inducing input Gaussian process classification
Pavel Izmailov, Dmitry Kropotov
Gaussian processes (GP) provide a prior over functions and allow finding complex regularities in data. Gaussian processes are successfully used for classification/regression proble…