Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent
arXiv:2002.04861
Abstract
We prove that two-layer (Leaky)ReLU networks initialized by e.g. the widely used method proposed by He et al. (2015) and trained using gradient descent on a least-squares loss are not universally consistent. Specifically, we describe a large class of one-dimensional data-generating distributions for which, with high probability, gradient descent only finds a bad local minimum of the optimization landscape, since it is unable to move the biases far away from their initialization at zero. It turns out that in these cases, the found network essentially performs linear regression even if the target function is non-linear. We further provide numerical evidence that this happens in practical situations, for some multi-dimensional distributions and that stochastic gradient descent exhibits similar behavior. We also provide empirical results on how the choice of initialization and optimizer can influence this behavior.
arXiv commentTo appear in Journal of Machine Learning Research (JMLR). Changes in v3: Added new Section 10 with extensive experimental evaluation. Code available at https://github.com/dholzmueller/nn_inconsistency
References in corpus (18)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- On Exact Computation with an Infinitely Wide Neural Net
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- On Lazy Training in Differentiable Programming
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
- How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
- Small nonlinearities in activation functions create bad local minima in neural networks
- Complexity of Training ReLU Neural Network
- Piecewise linear activations substantially shape the loss surfaces of neural networks
- Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
- A Sober Look at Neural Network Initializations
- Empirical Risk Minimization in the Interpolating Regime with Application to Neural Network Learning