A Field Guide to Federated Optimization

Abstract

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

References in corpus (105)

• Deep Learning with Differential Privacy
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Equality of Opportunity in Supervised Learning
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Reconciling modern machine learning practice and the bias-variance trade-off
• Federated Learning for Mobile Keyboard Prediction
• On the Convergence of FedAvg on Non-IID Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Robust Aggregation for Federated Learning
• Learning Differentially Private Recurrent Language Models
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Revisiting Distributed Synchronous SGD
• Optimal Distributed Online Prediction using Mini-Batches
• Federated Learning with Personalization Layers
• On First-Order Meta-Learning Algorithms
• Training Deep Nets with Sublinear Memory Cost
• FedMD: Heterogenous Federated Learning via Model Distillation
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Can You Really Backdoor Federated Learning?
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• A generic framework for privacy preserving deep learning
• Three Approaches for Personalization with Applications to Federated Learning
• Fairness in Machine Learning
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Federated Learning of a Mixture of Global and Local Models
• Sparsified SGD with Memory
• Distributed Coordinate Descent Method for Learning with Big Data
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• SecureBoost: A Lossless Federated Learning Framework
• Split learning for health: Distributed deep learning without sharing raw patient data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Robust Federated Learning in a Heterogeneous Environment
• Bayesian Nonparametric Federated Learning of Neural Networks
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Distributed Mean Estimation with Limited Communication
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• Federated Learning Based on Dynamic Regularization
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• AIDE: Fast and Communication Efficient Distributed Optimization
• Flower: A Friendly Federated Learning Research Framework
• Federated Learning Of Out-Of-Vocabulary Words
• Federated Learning with Matched Averaging
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Optimal Client Sampling for Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Parallel SGD: When does averaging help?
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• SparkNet: Training Deep Networks in Spark
• Federated Mutual Learning
• Salvaging Federated Learning by Local Adaptation
• Natural Compression for Distributed Deep Learning
• Scalable and Differentially Private Distributed Aggregation in the Shuffled Model
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• On Biased Compression for Distributed Learning
• Is Local SGD Better than Minibatch SGD?
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Hierarchically Fair Federated Learning
• Generative Models for Effective ML on Private, Decentralized Datasets
• Secure Byzantine-Robust Machine Learning
• Personalized Cross-Silo Federated Learning on Non-IID Data
• Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation
• Eavesdrop the Composition Proportion of Training Labels in Federated Learning
• Breaking the Communication-Privacy-Accuracy Trilemma
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Federated Learning Meets Multi-objective Optimization
• Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• Federated Reconstruction: Partially Local Federated Learning
• Distributed Learning for Cooperative Inference
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Distributionally Robust Federated Averaging
• On the Outsized Importance of Learning Rates in Local Update Methods
• Federated Residual Learning
• Parle: parallelizing stochastic gradient descent
• On the Utility of Gradient Compression in Distributed Training Systems
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• Data Poisoning Attacks on Federated Machine Learning
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspaces
• Error Compensated Distributed SGD Can Be Accelerated
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Federated Hierarchical Hybrid Networks for Clickbait Detection
• Distributed Optimization for Over-Parameterized Learning
• When does gradient descent with logistic loss find interpolating two-layer networks?
• On Large-Cohort Training for Federated Learning
• QuPeL: Quantized Personalization with Applications to Federated Learning

Cited by in corpus (42)

• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints
• Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning
• On-board Federated Learning for Satellite Clusters with Inter-Satellite Links
• Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Improving Fairness via Federated Learning
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
• FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy
• Differentially Private Federated Learning on Heterogeneous Data
• Edge-cloud Collaborative Learning with Federated and Centralized Features
• Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches
• Scaling Survival Analysis in Healthcare with Federated Survival Forests: A Comparative Study on Heart Failure and Breast Cancer Genomics
• Federated Survival Forests
• Accelerating Vertical Federated Learning
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• What Do We Mean by Generalization in Federated Learning?
• Efficient and Private Federated Learning with Partially Trainable Networks
• Network Shuffling: Privacy Amplification via Random Walks
• ScionFL: Efficient and Robust Secure Quantized Aggregation
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clients
• Tackling Dynamics in Federated Incremental Learning with Variational Embedding Rehearsal
• Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition
• Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
• Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• FedPrune: Towards Inclusive Federated Learning
• DP-REC: Private & Communication-Efficient Federated Learning
• A Communication-efficient Local Differentially Private Algorithm in Federated Optimization
• BOBA: Byzantine-Robust Federated Learning with Label Skewness
• Fairness-aware Federated Minimax Optimization with Convergence Guarantee
• Neural Tangent Kernel Empowered Federated Learning
• Linear Speedup in Personalized Collaborative Learning
• Iterated Vector Fields and Conservatism, with Applications to Federated Learning
• One-Shot Clustering for Federated Learning
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications