Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Abstract

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.

The theoretical review and analysis in this article draw heavily from arXiv:1405.4604 [cs.LG]

References in corpus (2)

• Theano: new features and speed improvements
• On the saddle point problem for non-convex optimization

Cited by in corpus (22)

• FitNets: Hints for Thin Deep Nets
• The Loss Surfaces of Multilayer Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Qualitatively characterizing neural network optimization problems
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Explorations on high dimensional landscapes
• Classification regions of deep neural networks
• Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
• Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks
• Convexified Convolutional Neural Networks
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• AdaDNNs: Adaptive Ensemble of Deep Neural Networks for Scene Text Recognition
• Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently
• Low-rank Bilinear Pooling for Fine-Grained Classification
• Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima
• Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
• Improved Deep Learning of Object Category using Pose Information
• Expanded Alternating Optimization of Nonconvex Functions with Applications to Matrix Factorization and Penalized Regression
• ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent
• Are Saddles Good Enough for Deep Learning?
• How regularization affects the critical points in linear networks
• On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience