Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study

Abstract

While first-order optimization methods such as stochastic gradient descent (SGD) are popular in machine learning (ML), they come with well-known deficiencies, including relatively-slow convergence, sensitivity to the settings of hyper-parameters such as learning rate, stagnation at high training errors, and difficulty in escaping flat regions and saddle points. These issues are particularly acute in highly non-convex settings such as those arising in neural networks. Motivated by this, there has been recent interest in second-order methods that aim to alleviate these shortcomings by capturing curvature information. In this paper, we report detailed empirical evaluations of a class of Newton-type methods, namely sub-sampled variants of trust region (TR) and adaptive regularization with cubics (ARC) algorithms, for non-convex ML problems. In doing so, we demonstrate that these methods not only can be computationally competitive with hand-tuned SGD with momentum, obtaining comparable or better generalization performance, but also they are highly robust to hyper-parameter settings. Further, in contrast to SGD with momentum, we show that the manner in which these Newton-type methods employ curvature information allows them to seamlessly escape flat regions and saddle points.

21 pages, 11 figures. Restructure the paper and add experiments

References in corpus (5)

• The Loss Surfaces of Multilayer Networks
• How to Escape Saddle Points Efficiently
• The Power of Normalization: Faster Evasion of Saddle Points
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Local minima in training of neural networks

Cited by in corpus (14)

• DynNet: Physics-based neural architecture design for linear and nonlinear structural response modeling and prediction
• PyHessian: Neural Networks Through the Lens of the Hessian
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• Inexact Non-Convex Newton-Type Methods
• Stochastic Variance-Reduced Cubic Regularized Newton Method
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
• Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient
• GPU Accelerated Sub-Sampled Newton's Method
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Newton-ADMM: A Distributed GPU-Accelerated Optimizer for Multiclass Classification Problems
• Bernstein Concentration Inequalities for Tensors via Einstein Products
• Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method
• An Adaptive Sample Size Trust-Region Method for Finite-Sum Minimization
• GO Hessian for Expectation-Based Objectives