Communication-Efficient Agnostic Federated Averaging

Abstract

In distributed learning settings such as federated learning, the training algorithm can be potentially biased towards different clients. Mohri et al. (2019) proposed a domain-agnostic learning algorithm, where the model is optimized for any target distribution formed by a mixture of the client distributions in order to overcome this bias. They further proposed an algorithm for the cross-silo federated learning setting, where the number of clients is small. We consider this problem in the cross-device setting, where the number of clients is much larger. We propose a communication-efficient distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg) to minimize the domain-agnostic objective proposed in Mohri et al. (2019), which is amenable to other private mechanisms such as secure aggregation. We highlight two types of naturally occurring domains in federated learning and argue that AgnosticFedAvg performs well on both. To demonstrate the practical effectiveness of AgnosticFedAvg, we report positive results for large-scale language modeling tasks in both simulation and live experiments, where the latter involves training language models for Spanish virtual keyboard for millions of user devices.

References in corpus (20)

• LSTM: A Search Space Odyssey
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Equality of Opportunity in Supervised Learning
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning for Mobile Keyboard Prediction
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• Adaptive Federated Optimization
• Federated Learning Based on Dynamic Regularization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
• Mitigating Bias in Federated Learning
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• FedJAX: Federated learning simulation with JAX
• Federated Residual Learning

Cited by in corpus (2)

• FedJAX: Federated learning simulation with JAX
• Unified Group Fairness on Federated Learning