Generalization of Quasi-Newton Methods: Application to Robust Symmetric Multisecant Updates
arXiv:2011.03358
Abstract
Quasi-Newton techniques approximate the Newton step by estimating the Hessian using the so-called secant equations. Some of these methods compute the Hessian using several secant equations but produce non-symmetric updates. Other quasi-Newton schemes, such as BFGS, enforce symmetry but cannot satisfy more than one secant equation. We propose a new type of quasi-Newton symmetric update using several secant equations in a least-squares sense. Our approach generalizes and unifies the design of quasi-Newton updates and satisfies provable robustness guarantees.
arXiv commentAISTATS 2021
References in corpus (9)
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- A Linearly-Convergent Stochastic L-BFGS Algorithm
- Global Convergence of Online Limited Memory BFGS
- Convergence rates of sub-sampled Newton methods
- Sub-sampled Newton Methods with Non-uniform Sampling
- A Progressive Batching L-BFGS Method for Machine Learning
- Globally Convergent Type-I Anderson Acceleration for Non-Smooth Fixed-Point Iterations
- Nonlinear Acceleration of Stochastic Algorithms
- Online Regularized Nonlinear Acceleration