On Second-order Optimization Methods for Federated Learning

Abstract

We consider federated learning (FL), where the training data is distributed across a large number of clients. The standard optimization method in this setting is Federated Averaging (FedAvg), which performs multiple local first-order optimization steps between communication rounds. In this work, we evaluate the performance of several second-order distributed methods with local steps in the FL setting which promise to have favorable convergence properties. We (i) show that FedAvg performs surprisingly well against its second-order competitors when evaluated under fair metrics (equal amount of local computations)-in contrast to the results of previous work. Based on our numerical study, we propose (ii) a novel variant that uses second-order local information for updates and a global line search to counteract the resulting local specificity.

arXiv commentICML 2021 Workshop "Beyond first-order methods in ML systems"

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• A Reliable Effective Terascale Linear Learning System
• Local SGD Converges Fast and Communicates Little
• AIDE: Fast and Communication Efficient Distributed Optimization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Is Local SGD Better than Minibatch SGD?
• DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization
• GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication
• Distributed estimation of the inverse Hessian by determinantal averaging
• LocalNewton: Reducing Communication Bottleneck for Distributed Learning