Implicit Regularization in ReLU Networks with the Square Loss

Abstract

Understanding the implicit regularization (or implicit bias) of gradient descent has recently been a very active research area. However, the implicit regularization in nonlinear neural networks is still poorly understood, especially for regression losses such as the square loss. Perhaps surprisingly, we prove that even for a single ReLU neuron, it is impossible to characterize the implicit regularization with the square loss by any explicit function of the model parameters (although on the positive side, we show it can be characterized approximately). For one hidden-layer networks, we prove a similar result, where in general it is impossible to characterize implicit regularization properties in this manner, except for the "balancedness" property identified in Du et al. [2018]. Our results suggest that a more general framework than the one considered so far may be needed to understand implicit regularization for nonlinear predictors, and provides some clues on what this framework should be.

arXiv commentSmall changes due to reviews

References in corpus (16)

• Understanding deep learning requires rethinking generalization
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Gradient descent aligns the layers of deep linear networks
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Risk and parameter convergence of logistic regression
• Kernel and Rich Regimes in Overparametrized Models
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Gradient Dynamics of Shallow Univariate ReLU Networks
• When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Learning a Single Neuron with Gradient Methods
• On implicit regularization: Morse functions and applications to matrix factorization
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study
• Limitations of Implicit Bias in Matrix Sensing: Initialization Rank Matters
• Gradient descent follows the regularization path for general losses

Cited by in corpus (5)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• Continuous vs. Discrete Optimization of Deep Neural Networks
• Limitations of Implicit Bias in Matrix Sensing: Initialization Rank Matters