What Do We Mean by Generalization in Federated Learning?
arXiv:2110.14216
Abstract
Federated learning data is drawn from a distribution of distributions: clients are drawn from a meta-distribution, and their data are drawn from local data distributions. Thus generalization studies in federated learning should separate performance gaps from unseen client data (out-of-sample gap) from performance gaps from unseen client distributions (participation gap). In this work, we propose a framework for disentangling these performance gaps. Using this framework, we observe and explain differences in behavior across natural and synthetic federated datasets, indicating that dataset synthesis strategy can be important for realistic simulations of generalization in federated learning. We propose a semantic synthesis strategy that enables realistic simulation without naturally-partitioned data. Informed by our findings, we call out community suggestions for future federated learning works.
Accepted to ICLR 2022. Code repository see https://bit.ly/fl-generalization
References in corpus (32)
- Auto-Encoding Variational Bayes
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning with Non-IID Data
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Ensemble Distillation for Robust Model Fusion in Federated Learning
- On First-Order Meta-Learning Algorithms
- EMNIST: an extension of MNIST to handwritten letters
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Adaptive Personalized Federated Learning
- LEAF: A Benchmark for Federated Settings
- Federated Meta-Learning with Fast Convergence and Efficient Communication
- Agnostic Federated Learning
- Federated Learning of a Mixture of Global and Local Models
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- A Field Guide to Federated Optimization
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
- The Non-IID Data Quagmire of Decentralized Machine Learning
- FedSplit: An algorithmic framework for fast federated optimization
- Salvaging Federated Learning by Local Adaptation
- Lower Bounds and Optimal Algorithms for Personalized Federated Learning
- On Biased Compression for Distributed Learning
- Optimal Gradient Compression for Distributed and Federated Learning
- Linearly Converging Error Compensated SGD
- On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization
- Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
- A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
- Federated Residual Learning
- Distributed Second Order Methods with Fast Rates and Compressed Communication