551 citations · 2.1k across the 36 of their papers we have counts for
6 papers · 1 filter
Second-order regression models exhibit progressive sharpening to the edge of stability
Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive…
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm
Lechao Xiao, Jeffrey Pennington
Although learning in high dimensions is commonly believed to suffer from the curse of dimensionality, modern machine learning methods often exhibit an astonishing power to tackle a…
Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling
Jiri Hron, Roman Novak, Jeffrey Pennington +1
We introduce repriorisation, a data-dependent reparameterisation which transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior v…
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
Courtney Paquette, Elliot Paquette, Ben Adlam +1
Stochastic gradient descent (SGD) is a pillar of modern machine learning, serving as the go-to optimization algorithm for a diverse array of problems. While the empirical success o…
Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties
Courtney Paquette, Elliot Paquette, Ben Adlam +1
We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random least squares pr…
Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
Lechao Xiao, Hong Hu, Theodor Misiakiewicz +2
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improve…