The Implicit Bias of Depth: How Incremental Learning Drives Generalization

Abstract

A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models. Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves. However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings. We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.

arXiv comment25 pages, 7 figures, published at the International Conference on Learning Representations (ICLR) 2020

References in corpus (6)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Convergence of Gradient Descent on Separable Data

Cited by in corpus (6)

• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• The loss landscape of deep linear neural networks: a second-order analysis
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
• The staircase property: How hierarchical structure can guide deep learning
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics