Distributionally Robust Federated Averaging

Abstract

In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can only be updated infrequently on the global stage. To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate the accumulation of history gradients of the mixing parameter. We analyze the convergence rate of DRFA in both convex-linear and nonconvex-linear settings. We also generalize the proposed idea to objectives with regularization on the mixture parameter and propose a proximal variant, dubbed as DRFA-Prox, with provable convergence rates. We also analyze an alternative optimization method for regularized cases in strongly-convex-strongly-concave and non-convex (under PL condition)-strongly-concave settings. To the best of our knowledge, this paper is the first to solve distributionally robust federated learning with reduced communication, and to analyze the efficiency of local descent methods on distributed minimax problems. We give corroborating experimental evidence for our theoretical results in federated learning settings.

Published in NeurIPS 2020: https://proceedings.neurips.cc/paper/2020/hash/ac450d10e166657ec8f93a1b65ca1b14-Abstract.html

Cited by in corpus (13)

• Ditto: Fair and Robust Federated Learning Through Personalization
• A Field Guide to Federated Optimization
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Unified Group Fairness on Federated Learning
• AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization
• Is Shapley Value fair? Improving Client Selection for Mavericks in Federated Learning
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Federating for Learning Group Fair Models
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization
• A Wasserstein Minimax Framework for Mixed Linear Regression