Implicit Gradient Regularization

Abstract

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.

Correction to formula A.14 in Appendix A.1 and update to the acknowledgments

References in corpus (15)

• Reconciling modern machine learning practice and the bias-variance trade-off
• On Exact Computation with an Infinitely Wide Neural Net
• Scaling description of generalization with number of parameters in deep learning
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• The large learning rate phase of deep learning: the catapult mechanism
• Generalization in Deep Networks: The Role of Distance from Initialization
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
• A type of generalization error induced by initialization in deep neural networks
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• SGD Implicitly Regularizes Generalization Error
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models

Cited by in corpus (2)

• Multilayer Lookahead: a Nested Version of Lookahead
• Variational Laplace for Bayesian neural networks