When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
arXiv:2102.04998
Abstract
We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.
References in corpus (26)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Characterizing Implicit Bias in Terms of Optimization Geometry
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- Recovery Guarantees for One-hidden-layer Neural Networks
- Gradient descent aligns the layers of deep linear networks
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- Benign overfitting in ridge regression
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
- Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
- Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
- A Corrective View of Neural Networks: Representation, Memorization and Learning
- Failures of model-dependent generalization bounds for least-norm interpolation
- Optimizing Mode Connectivity via Neuron Alignment
- When does gradient descent with logistic loss find interpolating two-layer networks?