Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy

Abstract

We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks". This is the simplest model displaying a transition between "kernel" and non-kernel ("rich" or "active") regimes. We show how the transition is controlled by the relationship between the initialization scale and how accurately we minimize the training loss. Our results indicate that some limit behaviors of gradient descent only kick in at ridiculous training accuracies (well beyond $10^{-100}$). Moreover, the implicit bias at reasonable initialization scales and training accuracies is more complex and not captured by these limits.

References in corpus (11)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Margins, Shrinkage, and Boosting
• A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

Cited by in corpus (4)

• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Understanding Deflation Process in Over-parametrized Tensor Decomposition
• Towards Understanding Learning in Neural Networks with Linear Teachers