Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
arXiv:2102.09743
Abstract
We investigate the optimization aspects of personalized Federated Learning (FL). We propose general optimizers that can be applied to numerous existing personalized FL objectives, specifically a tailored variant of Local SGD and variants of accelerated coordinate descent/accelerated SVRCD. By examining a general personalized objective capable of recovering many existing personalized FL objectives as special cases, we develop a comprehensive optimization theory applicable to a wide range of strongly convex personalized FL models in the literature. We showcase the practicality and/or optimality of our methods in terms of communication and local computation. Remarkably, our general optimization solvers and theory can recover the best-known communication and computation guarantees for addressing specific personalized FL objectives. Consequently, our proposed methods can serve as universal optimizers, rendering the design of task-specific optimizers unnecessary in many instances.
References in corpus (26)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning for Mobile Keyboard Prediction
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Federated Learning with Personalization Layers
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Personalized Federated Learning: A Meta-Learning Approach
- Adaptive Personalized Federated Learning
- Federated Meta-Learning with Fast Convergence and Efficient Communication
- Federated Learning of a Mixture of Global and Local Models
- Ditto: Fair and Robust Federated Learning Through Personalization
- Personalized Federated Learning with Moreau Envelopes
- On the Convergence of Local Descent Methods in Federated Learning
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
- Federated Multi-Task Learning
- Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling
- Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
- Lower Bounds and Optimal Algorithms for Personalized Federated Learning
- Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
- Real-Time Edge Intelligence in the Making: A Collaborative Learning Framework via Federated Meta-Learning
- Federated Residual Learning
- Distributed Stochastic Multi-Task Learning with Graph Regularization
- Local SGD: Unified Theory and New Efficient Methods
- Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models
- Variance Reduced Coordinate Descent with Acceleration: New Method With a Surprising Application to Finite-Sum Problems