Sub-Sampled Newton Methods II: Local Convergence Rates

Abstract

Many data-fitting applications require the solution of an optimization problem involving a sum of large number of functions of high dimensional parameter. Here, we consider the problem of minimizing a sum of $n$ functions over a convex constraint set $\mathcal{X}\subseteq {\mathbb{R}}^p$ where both $n$ and $p$ are large. In such problems, sub-sampling as a way to reduce $n$ can offer great amount of computational efficiency. Within the context of second order methods, we first give quantitative local convergence results for variants of Newton's method where the Hessian is uniformly sub-sampled. Using random matrix concentration inequalities, one can sub-sample in a way that the curvature information is preserved. Using such sub-sampling strategy, we establish locally Q-linear and Q-superlinear convergence rates. We also give additional convergence results for when the sub-sampled Hessian is regularized by modifying its spectrum or Levenberg-type regularization. Finally, in addition to Hessian sub-sampling, we consider sub-sampling the gradient as way to further reduce the computational complexity per iteration. We use approximate matrix multiplication results from randomized numerical linear algebra (RandNLA) to obtain the proper sampling strategy and we establish locally R-linear convergence rates. In such a setting, we also show that a very aggressive sample size increase results in a R-superlinearly convergent algorithm. While the sample size depends on the condition number of the problem, our convergence rates are problem-independent, i.e., they do not depend on the quantities related to the problem. Hence, our analysis here can be used to complement the results of our basic framework from the companion paper, [38], by exploring algorithmic trade-offs that are important in practice.

References in corpus (7)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Better Mini-Batch Algorithms via Accelerated Gradient Methods
• Convergence rates of sub-sampled Newton methods
• Sub-Sampled Newton Methods I: Globally Convergent Algorithms
• Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence
• Fast Stochastic Algorithms for SVD and PCA: Convergence Properties and Convexity
• Simultaneous Source for non-uniform data variance and missing data

Cited by in corpus (36)

• Sub-sampled Newton Methods with Non-uniform Sampling
• Sub-Sampled Newton Methods I: Globally Convergent Algorithms
• A Progressive Batching L-BFGS Method for Machine Learning
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• A Survey of Optimization Methods from a Machine Learning Perspective
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Multiplicative noise and heavy tails in stochastic optimization
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• Inexact Non-Convex Newton-Type Methods
• Stochastic Variance-Reduced Cubic Regularized Newton Method
• Robust Frequent Directions with Application in Online Learning
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
• An inexact subsampled proximal Newton-type method for large-scale machine learning
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Regularization by Denoising Sub-sampled Newton Method for Spectral CT Multi-Material Decomposition
• Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient
• Stochastic Second-Order Optimization via von Neumann Series
• Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations
• Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
• Newton-ADMM: A Distributed GPU-Accelerated Optimizer for Multiclass Classification Problems
• Convergence Analysis of Block Coordinate Algorithms with Determinantal Sampling
• GPU Accelerated Sub-Sampled Newton's Method
• LocalNewton: Reducing Communication Bottleneck for Distributed Learning
• Large Scale Empirical Risk Minimization via Truncated Adaptive Newton Method
• Differentiable Visual Computing
• A Bootstrap Method for Error Estimation in Randomized Matrix Multiplication
• Randomized Approach to Nonlinear Inversion Combining Simultaneous Random and Optimized Sources and Detectors
• Inefficiency of K-FAC for Large Batch Size Training
• Parallel Stochastic Newton Method
• Revisiting Sub-sampled Newton Methods
• Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization
• Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods
• SPAN: A Stochastic Projected Approximate Newton Method
• Iterative Teaching by Label Synthesis
• Do Subsampled Newton Methods Work for High-Dimensional Data?