Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks

Abstract

When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequently, this choice can be considered as an implicit regularization for the training of over-parametrized models. In this work, we push this idea further by studying the discrete gradient dynamics of the training of a two-layer linear network with the least-squares loss. Using a time rescaling, we show that, with a vanishing initialization and a small enough step size, this dynamics sequentially learns the solutions of a reduced-rank regression with a gradually increasing rank.

19 pages, to appear in NeurIPS 2019 proceedings

References in corpus (3)

• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Implicit Regularization in Deep Learning
• An analytic theory of generalization dynamics and transfer learning in deep linear networks

Cited by in corpus (11)

• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Implicit Regularization in Deep Matrix Factorization
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Regularized linear autoencoders recover the principal components, eventually
• An Unconstrained Layer-Peeled Perspective on Neural Collapse
• When Does Preconditioning Help or Hurt Generalization?
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Bregman Proximal Framework for Deep Linear Neural Networks
• Limitations of Implicit Bias in Matrix Sensing: Initialization Rank Matters
• Implicit Sparse Regularization: The Impact of Depth and Early Stopping
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks