On Connected Sublevel Sets in Deep Learning
arXiv:1901.07417
Abstract
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.
Accepted at ICML 2019. More discussions and visualizations are added
References in corpus (21)
- The Loss Surfaces of Multilayer Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Gradient Descent Finds Global Minima of Deep Neural Networks
- The loss surface of deep and wide neural networks
- Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
- Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
- Gradient descent aligns the layers of deep linear networks
- Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
- On the Computational Efficiency of Training Neural Networks
- Depth Creates No Bad Local Minima
- An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
- Diverse Neural Network Learns True Target Functions
- Global optimality conditions for deep neural networks
- Optimization Landscape and Expressivity of Deep CNNs
- Spurious Valleys in Two-layer Neural Network Optimization Landscapes
- Critical Points of Neural Networks: Analytical Forms and Landscape Properties
- Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
- Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Cited by in corpus (6)
- Optimization for deep learning: theory and algorithms
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
- Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
- Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
- Training Linear Neural Networks: Non-Local Convergence and Complexity Results