Randomized Quasi-Newton Updates are Linearly Convergent Matrix Inversion Algorithms

Abstract

We develop and analyze a broad family of stochastic/randomized algorithms for inverting a matrix. We also develop specialized variants maintaining symmetry or positive definiteness of the iterates. All methods in the family converge globally and linearly (i.e., the error decays exponentially), with explicit rates. In special cases, we obtain stochastic block variants of several quasi-Newton updates, including bad Broyden (BB), good Broyden (GB), Powell-symmetric-Broyden (PSB), Davidon-Fletcher-Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS). Ours are the first stochastic versions of these updates shown to converge to an inverse of a fixed matrix. Through a dual viewpoint we uncover a fundamental link between quasi-Newton updates and approximate inverse preconditioning. Further, we develop an adaptive variant of randomized block BFGS, where we modify the distribution underlying the stochasticity of the method throughout the iterative process to achieve faster convergence. By inverting several matrices from varied applications, we demonstrate that AdaRBFGS is highly competitive when compared to the well established Newton-Schulz and minimal residual methods. In particular, on large-scale problems our method outperforms the standard methods by orders of magnitude. Development of efficient methods for estimating the inverse of very large matrices is a much needed tool for preconditioning and variable metric optimization methods in the advent of the big data era.

arXiv comment42 pages, 6 figures, 2 tables

References in corpus (4)

• Stochastic Dual Ascent for Solving Linear Systems
• Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence
• Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares
• Action constrained quasi-Newton methods

Cited by in corpus (10)

• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Stochastic, Distributed and Federated Optimization for Machine Learning
• Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
• Privacy Preserving Randomized Gossip Algorithms
• Linearly Convergent Randomized Iterative Methods for Computing the Pseudoinverse
• Stochastic Block BFGS: Squeezing More Curvature out of Data
• Sketch and Project: Randomized Iterative Methods for Linear Systems and Inverting Matrices
• Stochastic Newton and Quasi-Newton Methods for Large Linear Least-squares Problems
• Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods
• A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation