563 citations · 891 across the 25 of their papers we have counts for
26 papers
Sharpness-Aware Minimization and the Edge of Stability
Philip M. Long, Peter L. Bartlett
Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size , the operator norm of the Hessian of the loss grows until…
Prediction, Learning, Uniform Convergence, and Scale-sensitive Dimensions
Peter L. Bartlett, Philip M. Long
We present a new general-purpose algorithm for learning classes of -valued functions in a generalization of the prediction model, and prove a general upper bound on the expe…
The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima
Peter L. Bartlett, Philip M. Long, Olivier Bousquet
We consider Sharpness-Aware Minimization (SAM), a gradient-based optimization method for deep networks that has exhibited performance improvements on image and language prediction…
Deep Linear Networks can Benignly Overfit when Shallow Ones Do
Niladri S. Chatterji, Philip M. Long
We bound the excess risk of interpolating deep linear networks trained using gradient flow. In a setting previously used to establish risk bounds for the minimum -norm inte…
The perils of being unhinged: On the accuracy of classifiers minimizing a noise-robust convex loss
Philip M. Long, Rocco A. Servedio
Van Rooyen et al. introduced a notion of convex loss functions being robust to random classification noise, and established that the "unhinged" loss function is robust in this sens…
Foolish Crowds Support Benign Overfitting
Niladri S. Chatterji, Philip M. Long
We prove a lower bound on the excess risk of sparse interpolating procedures for linear regression with Gaussian data in the overparameterized regime. We apply this result to obtai…
The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks
Niladri S. Chatterji, Philip M. Long, Peter L. Bartlett
The recent success of neural network models has shone light on a rather surprising statistical phenomenon: statistical models that perfectly fit noisy data can generalize well to u…
Properties of the After Kernel
Philip M. Long
The Neural Tangent Kernel (NTK) is the wide-network limit of a kernel defined using neural networks at initialization, whose embedding is the gradient of the output of the network…
When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
Niladri S. Chatterji, Philip M. Long, Peter L. Bartlett
We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysi…
When does gradient descent with logistic loss find interpolating two-layer networks?
Niladri S. Chatterji, Philip M. Long, Peter L. Bartlett
We study the training of finite-width two-layer smoothed ReLU networks for binary classification using the logistic loss. We show that gradient descent drives the training loss to…
Failures of model-dependent generalization bounds for least-norm interpolation
Peter L. Bartlett, Philip M. Long
We consider bounds on the generalization performance of the least-norm linear regressor, in the over-parameterized regime where it can interpolate the data. We describe a sense in…
Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime
Niladri S. Chatterji, Philip M. Long
We prove bounds on the population risk of the maximum margin algorithm for two-class linear classification. For linearly separable training data, the maximum margin algorithm has b…
On the Global Convergence of Training Deep Linear ResNets
Difan Zou, Philip M. Long, Quanquan Gu
We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training -hidden-layer linear residual networks (ResNets). We prove that for training…
Oracle Lower Bounds for Stochastic Gradient Sampling Algorithms
Niladri S. Chatterji, Peter L. Bartlett, Philip M. Long
We consider the problem of sampling from a strongly log-concave density in , and prove an information theoretic lower bound on the number of stochastic gradient queri…
Benign Overfitting in Linear Regression
Peter L. Bartlett, Philip M. Long, Gábor Lugosi +1
The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy…
Generalization bounds for deep convolutional neural networks
Philip M. Long, Hanie Sedghi
We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss an…
On the effect of the activation function on the distribution of hidden nodes in a deep network
Philip M. Long, Hanie Sedghi
We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases a…
Density estimation for shift-invariant multidimensional distributions
Anindya De, Philip M. Long, Rocco A. Servedio
We study density estimation for classes of shift-invariant distributions over . A multidimensional distribution is "shift-invariant" if, roughly speaking, it is close…
Learning Sums of Independent Random Variables with Sparse Collective Support
Anindya De, Philip M. Long, Rocco A. Servedio
We study the learnability of sums of independent integer random variables given a bound on the size of the union of their supports. For , a sum…
The Singular Values of Convolutional Layers
Hanie Sedghi, Vineet Gupta, Philip M. Long
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This charact…
Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
Peter L. Bartlett, Steven N. Evans, Philip M. Long
We show that any smooth bi-Lipschitz can be represented exactly as a composition of functions that are close to the identity in the sens…
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks
Peter L. Bartlett, David P. Helmbold, Philip M. Long
We analyze algorithms for approximating a function mapping to using deep linear neural networks, i.e. that learn a function parameterized by matrice…
Surprising properties of dropout in deep networks
David P. Helmbold, Philip M. Long
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more tradition…
On the Inductive Bias of Dropout
David P. Helmbold, Philip M. Long
Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout sho…