When does gradient descent with logistic loss find interpolating two-layer networks?
arXiv:2012.02409
Abstract
We study the training of finite-width two-layer smoothed ReLU networks for binary classification using the logistic loss. We show that gradient descent drives the training loss to zero if the initial loss is small enough. When the data satisfies certain cluster and separation conditions and the network is wide enough, we show that one step of gradient descent reduces the loss sufficiently that the first result applies.
References in corpus (25)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Characterizing Implicit Bias in Terms of Optimization Geometry
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- Recovery Guarantees for One-hidden-layer Neural Networks
- Gradient descent aligns the layers of deep linear networks
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- Benign overfitting in ridge regression
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
- Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
- A Corrective View of Neural Networks: Representation, Memorization and Learning
- Failures of model-dependent generalization bounds for least-norm interpolation
- Optimizing Mode Connectivity via Neuron Alignment
- When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
- On the proliferation of support vectors in high dimensions