Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Abstract

Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted $K$ asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.

References in corpus (18)

• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Federated Machine Learning: Concept and Applications
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Asynchronous Federated Optimization
• SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overhead
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• A generic framework for privacy preserving deep learning
• Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computing
• Staleness-aware Async-SGD for Distributed Deep Learning
• A Bi-layered Parallel Training Architecture for Large-scale Convolutional Neural Networks
• Adaptive Bound Optimization for Online Convex Optimization
• Variance Reduced Local SGD with Lower Communication Complexity
• Asynchronous Federated Learning with Differential Privacy for Edge Intelligence
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• More Iterations per Second, Same Quality -- Why Asynchronous Algorithms may Drastically Outperform Traditional Ones

Cited by in corpus (1)

• FedGSM: Efficient Federated Learning for LEO Constellations with Gradient Staleness Mitigation