Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process

Abstract

We consider networks, trained via stochastic gradient descent to minimize ${\ell }_2$ loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization term corresponding to the sum over the data points, of the squared ${\ell }_2$ norm of the gradient of the model with respect to the parameter vector, evaluated at each data point. This holds for networks of any connectivity, width, depth, and choice of activation function. We interpret this implicit regularization term for three simple settings: matrix sensing, two layer ReLU networks trained on one-dimensional data, and two layer networks with sigmoid activations trained on a single datapoint. For these settings, we show why this new and general implicit regularization effect drives the networks towards "simple" models.

References in corpus (9)

• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Adding noise to the input of a model trained with a regularized objective
• A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
• High probability generalization bounds for uniformly stable algorithms with nearly optimal rate
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Gradient Descent Quantizes ReLU Network Features

Cited by in corpus (9)

• On the Opportunities and Risks of Foundation Models
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Does Data Augmentation Improve Generalization in NLP?
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Label Noise SGD Provably Prefers Flat Global Minimizers
• On Sparsity in Overparametrised Shallow ReLU Networks
• Regularization by Misclassification in ReLU Neural Networks
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time