Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server

Abstract

This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss is strongly convex, we establish linear convergence rate, give explicit expressions for step-size choices that guarantee convergence to the optimum, and bound the associated convergence factors. The expressions have an explicit dependence on the degree of asynchrony and recover classical results under synchronous operation. Simulations and implementations on commercial compute clouds validate our findings.

10 pages, 3 figures

References in corpus (1)

• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

Cited by in corpus (7)

• Fast Federated Learning in the Presence of Arbitrary Device Unavailability
• Advances in Asynchronous Parallel and Distributed Optimization
• DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization
• A Stronger Convergence Result on the Proximal Incremental Aggregated Gradient Method
• Inertial Proximal Incremental Aggregated Gradient Method
• On the Convergence of Primal-Dual Proximal Incremental Aggregated Gradient Algorithms
• Linear convergence of random dual coordinate incremental aggregated gradient methods