Linear Speedup in Personalized Collaborative Learning
arXiv:2111.05968
Abstract
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user). In this work, we formalize the personalized collaborative learning problem as a stochastic optimization of a task 0 while giving access to N related but different tasks 1,..., N. We provide convergence guarantees for two algorithms in this setting -- a popular collaboration method known as weighted gradient averaging, and a novel bias correction method -- and explore conditions under which we can achieve linear speedup w.r.t. the number of auxiliary tasks N. Further, we also empirically study their performance confirming our theoretical insights.
References in corpus (23)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Personalized Federated Learning: A Meta-Learning Approach
- Adaptive Personalized Federated Learning
- Three Approaches for Personalization with Applications to Federated Learning
- No More Pesky Learning Rates
- Agnostic Federated Learning
- A Field Guide to Federated Optimization
- AIDE: Fast and Communication Efficient Distributed Optimization
- The Benefit of Multitask Representation Learning
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
- Salvaging Federated Learning by Local Adaptation
- On the Theory of Transfer Learning: The Importance of Task Diversity
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Learning from History for Byzantine Robust Optimization
- Byzantine-Tolerant Machine Learning
- On the Convergence of SGD with Biased Gradients
- How Does the Task Landscape Affect MAML Performance?
- Provable Adaptation across Multiway Domains via Representation Learning
- WAFFLE: Weighted Averaging for Personalized Federated Learning
- Sample Efficient Linear Meta-Learning by Alternating Minimization