Local Adaptivity in Federated Learning: Convergence and Consistency

Abstract

The federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server. Recently, adaptive optimization methods such as AdaGrad have been studied for server updates. However, the effect of using adaptive optimization methods for local updates at clients is not yet understood. We show in both theory and practice that while local adaptive methods can accelerate convergence, they can cause a non-vanishing solution bias, where the final converged solution may be different from the stationary point of the global objective function. We propose correction techniques to overcome this inconsistency and complement the local adaptive methods for FL. Extensive experiments on realistic federated training tasks show that the proposed algorithms can achieve faster convergence and higher test accuracy than the baselines without local adaptivity.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• On the Convergence of Adam and Beyond
• Federated Learning for Mobile Keyboard Prediction
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• New insights and perspectives on the natural gradient method
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Adaptive Federated Optimization
• Optimal Client Sampling for Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• FedSplit: An algorithmic framework for fast federated optimization
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Unified Optimal Analysis of the (Stochastic) Gradient Method
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• From Local SGD to Local Fixed-Point Methods for Federated Learning

Cited by in corpus (5)

• A Field Guide to Federated Optimization
• Federated Learning for Internet of Things: A Federated Learning Framework for On-device Anomaly Data Detection
• Federated brain tumor segmentation: an extensive benchmark
• Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data
• TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clients