430 citations · 643 across the 26 of their papers we have counts for
12 papers · 1 filter
Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysis
Yufan Li, Subhabrata Sen, Ben Adlam
In the transfer learning paradigm models learn useful representations (or features) during a data-rich pretraining stage, and then use the pretrained representation to improve mode…
Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Ben Adlam, Jaehoon Lee, Shreyas Padhy +2
Neural kernels have drastically increased performance on diverse and nonstandard data modalities but require significantly more compute, which previously limited their application…
Ensembling over Classifiers: a Bias-Variance Perspective
Neha Gupta, Jamie Smith, Ben Adlam +1
Ensembles are a straightforward, remarkably effective method for improving the accuracy,calibration, and robustness of models on classification tasks; yet, the reasons that underli…
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
Courtney Paquette, Elliot Paquette, Ben Adlam +1
Stochastic gradient descent (SGD) is a pillar of modern machine learning, serving as the go-to optimization algorithm for a diverse array of problems. While the empirical success o…
Understanding the bias-variance tradeoff of Bregman divergences
Ben Adlam, Neha Gupta, Zelda Mariet +1
This paper builds upon the work of Pfau (2013), which generalized the bias variance tradeoff to any Bregman divergence loss function. Pfau (2013) showed that for Bregman divergence…
Covariate Shift in High-Dimensional Random Feature Regression
Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input distributions of the trainin…
Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
Ben Adlam, Jeffrey Pennington
Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exh…
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
Ben Adlam, Jaehoon Lee, Lechao Xiao +2
Modern deep learning models have achieved great success in predictive accuracy for many data modalities. However, their application to many real-world tasks is restricted by poor u…
The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
Ben Adlam, Jeffrey Pennington
Modern deep learning models employ considerably more parameters than required to fit the training data. Whereas conventional statistical wisdom suggests such models should drastica…
Cold Posteriors and Aleatoric Uncertainty
Ben Adlam, Jasper Snoek, Samuel L. Smith
Recent work has observed that one can outperform exact inference in Bayesian neural networks by tuning the "temperature" of the posterior on a validation set (the "cold posterior"…
A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
Ben Adlam, Jake Levinson, Jeffrey Pennington
One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, of…
Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks
Ben Adlam, Charles Weill, Amol Kapoor
We investigate under and overfitting in Generative Adversarial Networks (GANs), using discriminators unseen by the generator to measure generalization. We find that the model capac…