Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
arXiv:2002.04763
Abstract
For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We give the locations and losses of differentiable local minima, and show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activation pattern of hidden neurons and network size. Furthermore, we give necessary and sufficient conditions for the existence of saddle points as well as non-differentiable local minima, and their locations if they exist.
References in corpus (29)
- The Loss Surfaces of Multilayer Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Qualitatively characterizing neural network optimization problems
- Essentially No Barriers in Neural Network Energy Landscape
- On the Power of Over-parametrization in Neural Networks with Quadratic Activation
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Optimization for deep learning: theory and algorithms
- Learning One-hidden-layer Neural Networks with Landscape Design
- Depth Creates No Bad Local Minima
- Identity Matters in Deep Learning
- Depth with Nonlinearity Creates No Bad Local Minima in ResNets
- The Landscape of Empirical Risk for Non-convex Losses
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Theory II: Landscape of the Empirical Risk in Deep Learning
- Understanding the Loss Surface of Neural Networks for Binary Classification
- Spurious Valleys in Two-layer Neural Network Optimization Landscapes
- Adding One Neuron Can Eliminate All Bad Local Minima
- Elimination of All Bad Local Minima in Deep Learning
- Are ResNets Provably Better than Linear Predictors?
- Local minima in training of neural networks
- Critical Points of Neural Networks: Analytical Forms and Landscape Properties
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Small nonlinearities in activation functions create bad local minima in neural networks
- Porcupine Neural Networks: (Almost) All Local Optima are Global
- Depth creates no more spurious local minima
- Piecewise Strong Convexity of Neural Networks
- Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 2: Experiments and Analysis