Federated Learning with Buffered Asynchronous Aggregation

Abstract

Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL - synchronized aggregation of client updates in FL - cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, which could result in an undesirable level of privacy for the system. To address these concerns, we propose a novel buffered asynchronous aggregation method, FedBuff, that is agnostic to the choice of optimizer, and combines the best properties of synchronous and asynchronous FL. We empirically demonstrate that FedBuff is 3.3x more efficient than synchronous FL and up to 2.5x more efficient than asynchronous FL, while being compatible with privacy-preserving technologies such as Secure Aggregation and differential privacy. We provide theoretical convergence guarantees in a smooth non-convex setting. Finally, we show that under differentially private training, FedBuff can outperform FedAvgM at low privacy settings and achieve the same utility for higher privacy settings.

arXiv commentAccepted at AISTATS 2022. Previously accepted at FL-ICML 2021

References in corpus (31)

• Deep Learning with Differential Privacy
• Practical Bayesian Optimization of Machine Learning Algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Federated Learning for Mobile Keyboard Prediction
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Federated Optimization in Heterogeneous Networks
• Asynchronous Federated Optimization
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• LEAF: A Benchmark for Federated Settings
• Three Factors Influencing Minima in SGD
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Practical Secure Aggregation for Federated Learning on User-Held Data
• On the Convergence of Local Descent Methods in Federated Learning
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Measuring the Effects of Data Parallelism on Neural Network Training
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Adaptive Federated Optimization
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise

Cited by in corpus (7)

• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• Papaya: Practical, Private, and Scalable Federated Learning
• Secure Aggregation for Buffered Asynchronous Federated Learning
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Boosting Asynchronous Decentralized Learning with Model Fragmentation
• Accelerating MoE Model Inference with Expert Sharding
• Practical Federated Learning without a Server