Local minima in training of neural networks

Abstract

There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under very strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Explaining and Harnessing Adversarial Examples
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Asynchronous Methods for Deep Reinforcement Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• Rethinking the Inception Architecture for Computer Vision
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Explorations on high dimensional landscapes

Cited by in corpus (22)

• Visualizing the Loss Landscape of Neural Nets
• Depth Creates No Bad Local Minima
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Inexact Non-Convex Newton-Type Methods
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• Symmetry & critical points for a model shallow neural network
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Are deep ResNets provably better than linear predictors?
• Learning Dynamics of Linear Denoising Autoencoders
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Deep Semi-Random Features for Nonlinear Function Approximation
• Critical Percolation as a Framework to Analyze the Training of Deep Networks
• A Generative Neural Network Framework for Automated Software Testing
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations