A Field Guide to Federated Optimization

Abstract

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

References in corpus (39)

• Equality of Opportunity in Supervised Learning
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Revisiting Distributed Synchronous SGD
• Federated Learning with Personalization Layers
• FedMD: Heterogenous Federated Learning via Model Distillation
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• Can You Really Backdoor Federated Learning?
• Fairness in Machine Learning
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Bayesian Nonparametric Federated Learning of Neural Networks
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• Federated Learning Based on Dynamic Regularization
• AIDE: Fast and Communication Efficient Distributed Optimization
• Federated Learning Of Out-Of-Vocabulary Words
• Federated Learning with Matched Averaging
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Parallel SGD: When does averaging help?
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Federated Mutual Learning
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Generative Models for Effective ML on Private, Decentralized Datasets
• Hierarchically Fair Federated Learning
• Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation
• Eavesdrop the Composition Proportion of Training Labels in Federated Learning
• Local Adaptivity in Federated Learning: Convergence and Consistency
• On the Outsized Importance of Learning Rates in Local Update Methods
• Distributionally Robust Federated Averaging
• Parle: parallelizing stochastic gradient descent
• Data Poisoning Attacks on Federated Machine Learning
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Error Compensated Distributed SGD Can Be Accelerated
• Federated Hierarchical Hybrid Networks for Clickbait Detection
• Distributed Optimization for Over-Parameterized Learning
• On Large-Cohort Training for Federated Learning
• QuPeL: Quantized Personalization with Applications to Federated Learning

Cited by in corpus (6)

• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Efficient and Private Federated Learning with Partially Trainable Networks
• Tackling Dynamics in Federated Incremental Learning with Variational Embedding Rehearsal
• FedPrune: Towards Inclusive Federated Learning