A Unifying View on Implicit Bias in Training Linear Neural Networks

Abstract

We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the network. For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the ${\ell }_{2/L}$ max-margin problem in a "transformed" input space defined by the network. For underdetermined regression, we prove that gradient flow finds a global minimum which minimizes a norm-like function that interpolates between weighted ${\ell }_1$ and ${\ell }_2$ norms in the transformed input space. Our theorems subsume existing results in the literature while removing standard convergence assumptions. We also provide experiments that corroborate our analysis.

arXiv comment38 pages, 7 figures. Revision after ICLR 2021 camera-ready version. Figure 2 newly added, theorem statements revised, including correction of Theorem 2

References in corpus (15)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
• Global Convergence of Gradient Descent for Deep Linear Residual Networks

Cited by in corpus (5)

• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Implicit Regularization in ReLU Networks with the Square Loss
• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• Rethinking Gauss-Newton for learning over-parameterized models
• Training Efficiency and Robustness in Deep Learning