DINO: Distributed Newton-Type Optimization Method

Abstract

We present a novel communication-efficient Newton-type algorithm for finite-sum optimization over a distributed computing environment. Our method, named DINO, overcomes both theoretical and practical shortcomings of similar existing methods. Under minimal assumptions, we guarantee global sub-linear convergence of DINO to a first-order stationary point for general non-convex functions and arbitrary data distribution over the network. Furthermore, for functions satisfying Polyak-Lojasiewicz (PL) inequality, we show that DINO enjoys a linear convergence rate. Our proposed algorithm is practically parameter free, in that it will converge regardless of the selected hyper-parameters, which are easy to tune. Additionally, its sub-problems are simple linear least-squares, for which efficient solvers exist. Numerical simulations demonstrate the efficiency of DINO as compared with similar alternatives.

arXiv comment16 pages

References in corpus (5)

• Revisiting Distributed Synchronous SGD
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• AIDE: Fast and Communication Efficient Distributed Optimization
• On exponential convergence of SGD in non-convex over-parametrized learning
• DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization