On the loss landscape of a class of deep neural networks with no bad local valleys

Abstract

We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily close to zero. This implies that these networks have no sub-optimal strict local minima.

arXiv commentAccepted at ICLR 2019

References in corpus (21)

• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Qualitatively characterizing neural network optimization problems
• The loss surface of deep and wide neural networks
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Recovery Guarantees for One-hidden-layer Neural Networks
• Learning ReLUs via Gradient Descent
• On the Computational Efficiency of Training Neural Networks
• Depth Creates No Bad Local Minima
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Failures of Gradient-Based Deep Learning
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Global optimality conditions for deep neural networks
• Optimization Landscape and Expressivity of Deep CNNs
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Complex powers of analytic functions and meromorphic renormalization in QFT
• Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex

Cited by in corpus (27)

• Optimization for deep learning: theory and algorithms
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• The Global Landscape of Neural Networks: An Overview
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Are deep ResNets provably better than linear predictors?
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• Neural Networks with Complex-Valued Weights Have No Spurious Local Minima
• Achieving Small Test Error in Mildly Overparameterized Neural Networks
• Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data
• Machine Learning and System Identification for Estimation in Physical Systems
• Theoretical Exploration of Flexible Transmitter Model
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points