FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers

Abstract

Federated learning (FL) involves training a model over massive distributed devices, while keeping the training data localized. This form of collaborative learning exposes new tradeoffs among model convergence speed, model accuracy, balance across clients, and communication cost, with new challenges including: (1) straggler problem, where the clients lag due to data or (computing and network) resource heterogeneity, and (2) communication bottleneck, where a large number of clients communicate their local updates to a central server and bottleneck the server. Many existing FL methods focus on optimizing along only one dimension of the tradeoff space. Existing solutions use asynchronous model updating or tiering-based synchronous mechanisms to tackle the straggler problem. However, the asynchronous methods can easily create a network communication bottleneck, while tiering may introduce biases as tiering favors faster tiers with shorter response latencies. To address these issues, we present FedAT, a novel Federated learning method with Asynchronous Tiers under Non-i.i.d. data. FedAT synergistically combines synchronous intra-tier training and asynchronous cross-tier training. By bridging the synchronous and asynchronous training through tiering, FedAT minimizes the straggler effect with improved convergence speed and test accuracy. FedAT uses a straggler-aware, weighted aggregation heuristic to steer and balance the training for further accuracy improvement. FedAT compresses the uplink and downlink communications using an efficient, polyline-encoding-based compression algorithm, therefore minimizing the communication cost. Results show that FedAT improves the prediction performance by up to 21.09%, and reduces the communication cost by up to 8.5x, compared to state-of-the-art FL methods.

arXiv commentSC21

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• TensorFlow: A system for large-scale machine learning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• Towards Federated Learning at Scale: System Design
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Federated Optimization in Heterogeneous Networks
• Asynchronous Federated Optimization
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• LEAF: A Benchmark for Federated Settings
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• An Exact Quantized Decentralized Gradient Descent Algorithm
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Federated Multi-Task Learning
• Deep learning with Elastic Averaging SGD
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling

Cited by in corpus (7)

• Ground-Assisted Federated Learning in LEO Satellite Constellations
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation
• Federated Learning with Buffered Asynchronous Aggregation
• Fast-Convergent Federated Learning with Adaptive Weighting
• Device Scheduling and Update Aggregation Policies for Asynchronous Federated Learning
• Asynchronous Federated Learning for Sensor Data with Concept Drift