On implicit regularization: Morse functions and applications to matrix factorization

Abstract

In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions. The key contributions are a new criterion for implicit regularization---a leading contender to explain the generalization power of deep models such as neural networks---and a general blueprint to study it. We apply these techniques to settle a conjecture on implicit regularization in matrix factorization.

References in corpus (2)

• Geometry of Optimization and Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Matrix Factorization

Cited by in corpus (5)

• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization
• Implicit Regularization in ReLU Networks with the Square Loss
• On Margin Maximization in Linear and ReLU Networks