Implicit bias of deep linear networks in the large learning rate phase
arXiv:2011.12547
Abstract
Most theoretical studies explaining the regularization effect in deep learning have only focused on gradient descent with a sufficient small learning rate or even gradient flow (infinitesimal learning rate). Such researches, however, have neglected a reasonably large learning rate applied in most practical applications. In this work, we characterize the implicit bias effect of deep linear networks for binary classification using the logistic loss in the large learning rate regime, inspired by the seminal work by Lewkowycz et al. [26] in a regression setting with squared loss. They found a learning rate regime with a large stepsize named the catapult phase, where the loss grows at the early stage of training and eventually converges to a minimum that is flatter than those found in the small learning rate regime. We claim that depending on the separation conditions of data, the gradient descent iterates will converge to a flatter minimum in the catapult phase. We rigorously prove this claim under the assumption of degenerate data by overcoming the difficulty of the non-constant Hessian of logistic loss and further characterize the behavior of loss and Hessian for non-separable data. Finally, we demonstrate that flatter minima in the space spanned by non-separable data along with the learning rate in the catapult phase can lead to better generalization empirically.
arXiv comment19 pages, 7 figures
References in corpus (23)
- Wide Residual Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- On Lazy Training in Differentiable Programming
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Characterizing Implicit Bias in Terms of Optimization Geometry
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Fantastic Generalization Measures and Where to Find Them
- Gradient descent aligns the layers of deep linear networks
- Enhanced Convolutional Neural Tangent Kernels
- Implicit Regularization in Deep Learning May Not Be Explainable by Norms
- The large learning rate phase of deep learning: the catapult mechanism
- Tensor Programs II: Neural Tangent Kernel for Any Architecture
- Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
- The Implicit Regularization of Stochastic Gradient Flow for Least Squares
- Convergence of Gradient Descent on Separable Data
- Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
- Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
- On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
- The Two Regimes of Deep Network Training
- Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
- Efficient Computation of Hessian Matrices in TensorFlow
- On Alignment in Deep Linear Neural Networks