Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
arXiv:1708.07827
Abstract
While first-order optimization methods such as stochastic gradient descent (SGD) are popular in machine learning (ML), they come with well-known deficiencies, including relatively-slow convergence, sensitivity to the settings of hyper-parameters such as learning rate, stagnation at high training errors, and difficulty in escaping flat regions and saddle points. These issues are particularly acute in highly non-convex settings such as those arising in neural networks. Motivated by this, there has been recent interest in second-order methods that aim to alleviate these shortcomings by capturing curvature information. In this paper, we report detailed empirical evaluations of a class of Newton-type methods, namely sub-sampled variants of trust region (TR) and adaptive regularization with cubics (ARC) algorithms, for non-convex ML problems. In doing so, we demonstrate that these methods not only can be computationally competitive with hand-tuned SGD with momentum, obtaining comparable or better generalization performance, but also they are highly robust to hyper-parameter settings. Further, in contrast to SGD with momentum, we show that the manner in which these Newton-type methods employ curvature information allows them to seamlessly escape flat regions and saddle points.
arXiv comment21 pages, 11 figures. Restructure the paper and add experiments
References in corpus (10)
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- The Loss Surfaces of Multilayer Networks
- How to Escape Saddle Points Efficiently
- When Are Nonconvex Problems Not Scary?
- The Power of Normalization: Faster Evasion of Saddle Points
- A Second-Order Cone Based Approach for Solving the Trust Region Subproblem and Its Variants
- Sub-Sampled Newton Methods I: Globally Convergent Algorithms
- Sub-Sampled Newton Methods II: Local Convergence Rates
- Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
- Local minima in training of neural networks
Cited by in corpus (14)
- DynNet: Physics-based neural architecture design for linear and nonlinear structural response modeling and prediction
- PyHessian: Neural Networks Through the Lens of the Hessian
- GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
- Inexact Non-Convex Newton-Type Methods
- Stochastic Variance-Reduced Cubic Regularized Newton Method
- Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
- Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient
- GPU Accelerated Sub-Sampled Newton's Method
- Parameter Re-Initialization through Cyclical Batch Size Schedules
- Newton-ADMM: A Distributed GPU-Accelerated Optimizer for Multiclass Classification Problems
- Bernstein Concentration Inequalities for Tensors via Einstein Products
- Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method
- An Adaptive Sample Size Trust-Region Method for Finite-Sum Minimization
- GO Hessian for Expectation-Based Objectives