Device Heterogeneity in Federated Learning: A Superquantile Approach

Abstract

We propose a federated learning framework to handle heterogeneous client devices which do not conform to the population data distribution. The approach hinges upon a parameterized superquantile-based objective, where the parameter ranges over levels of conformity. We present an optimization algorithm and establish its convergence to a stationary point. We show how to practically implement it using secure aggregation by interleaving iterations of the usual federated averaging method with device filtering. We conclude with numerical experiments on neural networks as well as linear models on tasks from computer vision and natural language processing.

References in corpus (19)

• Towards Federated Learning at Scale: System Design
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• EMNIST: an extension of MNIST to handwritten letters
• UVeQFed: Universal Vector Quantization for Federated Learning
• Agnostic Federated Learning
• Over-the-Air Federated Learning from Heterogeneous Data
• Personalized Federated Learning with Moreau Envelopes
• Federated Learning: A Signal Processing Perspective
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Bayesian Nonparametric Federated Learning of Neural Networks
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Exploiting Shared Representations for Personalized Federated Learning
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Large-Scale Methods for Distributionally Robust Optimization
• The Skellam Mechanism for Differentially Private Federated Learning
• Federated Learning with Partial Model Personalization
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
• A Smoother Way to Train Structured Prediction Models