On the Efficiency of Random Permutation for ADMM and Coordinate Descent

Abstract

Random permutation is observed to be powerful for optimization algorithms: for multi-block ADMM (alternating direction method of multipliers), while the classical cyclic version divergence, the randomly permuted version converges in practice; for BCD (block coordinate descent), the randomly permuted version is typically faster than other versions. In this paper, we provide strong theoretical evidence that random permutation has positive effects on ADMM and BCD, by analyzing randomly permuted ADMM (RP-ADMM) for solving linear systems of equations, and randomly permuted BCD (RP-BCD) for solving unconstrained quadratic problems. First, we prove that RP-ADMM converges in expectation for solving systems of linear equations. The key technical result is that the spectrum of the expected update matrix of RP-BCD lies in $(-1/3,1)$, instead of the typical range $(-1,1)$. Second, we establish expected convergence rates of RP-ADMM for solving linear sytems and RP-BCD for solving unconstrained quadratic problems. This expected rate of RP-BCD is $O(n)$ times better than the worst-case rate of cyclic BCD, thus establishing a gap of at least $O(n)$ between RP-BCD and cyclic BCD. To analyze RP-BCD, we propose a conjecture of a new matrix AM-GM (algebraic mean-geometric mean) inequality, and prove a weaker version of it.

53 pages. This version is a strengthened version of a previous technical report "On the expected convergence of randomly permuted ADMM" appeared on arxiv on April 2015, with several new results, mainly the ones on the expected convergence rate of RP-CD and RP-ADMM. Also proposed Bernoulli-randomized ADMM

References in corpus (8)

• Why Random Reshuffling Beats Stochastic Gradient Descent
• A Convergent 3-Block Semi-Proximal Alternating Direction Method of Multipliers for Conic Programming with $4$-Type of Constraints
• Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences
• Worst-case Complexity of Cyclic Coordinate Descent: $O(n^2)$ Gap with Randomized Version
• On the Sublinear Convergence Rate of Multi-Block ADMM
• Improved Iteration Complexity Bounds of Cyclic Block Coordinate Descent for Convex Problems
• On the Global Linear Convergence of the ADMM with Multi-Block Variables
• Iteration Complexity Analysis of Multi-Block ADMM for a Family of Convex Minimization without Strong Convexity

Cited by in corpus (12)

• Guaranteed Matrix Completion via Non-convex Factorization
• Global Convergence of ADMM in Nonconvex Nonsmooth Optimization
• Convergence of multi-block Bregman ADMM for nonconvex composite problems
• Worst-case Complexity of Cyclic Coordinate Descent: $O(n^2)$ Gap with Randomized Version
• Distributed VNF Scaling in Large-scale Datacenters: An ADMM-based Approach
• Randomized Primal-Dual Proximal Block Coordinate Updates
• Extended ADMM and BCD for Nonseparable Convex Minimization Models with Quadratic Coupling Terms: Convergence Analysis and Insights
• A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints
• Sampling-Based Methods for Multi-Block Optimization Problems over Transport Polytopes
• Random constraint sampling and duality for convex optimization
• Understanding Limitation of Two Symmetrized Orders by Worst-case Complexity
• A Derandomized Algorithm for RP-ADMM with Symmetric Gauss-Seidel Method