Compositional federated learning: Applications in distributionally robust averaging and meta learning

Abstract

In the paper, we propose an effective and efficient Compositional Federated Learning (ComFedL) algorithm for solving a new compositional Federated Learning (FL) framework, which frequently appears in many data mining and machine learning problems with a hierarchical structure such as distributionally robust FL and model-agnostic meta learning (MAML). Moreover, we study the convergence analysis of our ComFedL algorithm under some mild conditions, and prove that it achieves a convergence rate of $O(\frac{1}{\sqrt{T}})$, where $T$ denotes the number of iteration. To the best of our knowledge, our new Compositional FL framework is the first work to bridge federated learning with composition stochastic optimization. In particular, we first transform the distributionally robust FL (i.e., a minimax optimization problem) into a simple composition optimization problem by using KL divergence regularization. At the same time, we also first transform the distribution-agnostic MAML problem (i.e., a minimax optimization problem) into a simple yet effective composition optimization problem. Finally, we apply two popular machine learning tasks, i.e., distributionally robust FL and MAML to demonstrate the effectiveness of our algorithm.

arXiv comment10 pages, 6 figures

References in corpus (31)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Agnostic Federated Learning
• Personalized Federated Learning with Moreau Envelopes
• Adaptive Federated Optimization
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Online Meta-Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
• ES-MAML: Simple Hessian-Free Meta Learning
• On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
• Provable Guarantees for Gradient-Based Meta-Learning
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Task-Robust Model-Agnostic Meta-Learning
• Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularization
• Distributionally Robust Federated Averaging
• Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient
• Compositional ADAM: An Adaptive Compositional Solver