Randomized Iterative Methods for Linear Systems

Abstract

We develop a novel, fundamental and surprisingly simple randomized iterative method for solving consistent linear systems. Our method has six different but equivalent interpretations: sketch-and-project, constrain-and-approximate, random intersect, random linear solve, random update and random fixed point. By varying its two parameters$-$a positive definite matrix (defining geometry), and a random matrix (sampled in an independently and identically distributed fashion in each iteration)$-$we recover a comprehensive array of well-known algorithms as special cases, including the randomized Kaczmarz method, randomized Newton method, randomized coordinate descent method and random Gaussian pursuit. We naturally also obtain variants of all these methods using blocks and importance sampling. However, our method allows for a much wider selection of these two parameters, which leads to a number of new specific methods. We prove exponential convergence of the expected norm of the error in a single theorem, from which existing complexity results for known variants can be obtained. However, we also give an exact formula for the evolution of the expected iterates, which allows us to give lower bounds on the convergence rate.

31 pages, 10 figures, 2 tables

Cited by in corpus (24)

• Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
• Preasymptotic Convergence of Randomized Kaczmarz Method
• Simultaneous imaging of hard and soft biological tissues in a low-field dental MRI scanner
• A probabilistic generative model for semi-supervised training of coarse-grained surrogates and enforcing physical constraints through virtual observables
• A literature survey of matrix methods for data science
• Randomized Row and Column Iterative Methods with a Quantum Computer
• Adaptively Sketched Bregman Projection Methods for Linear Systems
• A deterministic Kaczmarz algorithm for solving linear systems
• On the Polyak momentum variants of the greedy deterministic single and multiple row-action methods
• Prepolarized MRI of Hard Tissues and Solid-State Matter
• Fast and Secure Distributed Nonnegative Matrix Factorization
• Sparse matrix factorizations for fast linear solvers with application to Laplacian systems
• MaRGA: a Graphical and Application Interface for MaRCoS
• On the relaxed greedy deterministic row and column iterative methods
• Sampled Limited Memory Methods for Massive Linear Inverse Problems
• A Sketched Finite Element Method for Elliptic Models
• An accelerated randomized Bregman-Kaczmarz method for strongly convex linearly constraint optimization
• On the Convergence of Randomized and Greedy Relaxation Schemes for Solving Nonsingular Linear Systems of Equations
• On global randomized block Kaczmarz algorithm for solving large-scale matrix equations
• On the extended randomized multiple row method for solving linear least-squares problems
• Convergence of Regularization Parameters for Solutions Using the Filtered Truncated Singular Value Decomposition
• PLSS: A Projected Linear Systems Solver
• Stochastic Iterative Methods for Online Rank Aggregation from Pairwise Comparisons
• Ballistic Convergence in Hit-and-Run Monte Carlo and a Coordinate-free Randomized Kaczmarz Algorithm