Device Heterogeneity in Federated Learning: A Superquantile Approach

Abstract

We propose a federated learning framework to handle heterogeneous client devices which do not conform to the population data distribution. The approach hinges upon a parameterized superquantile-based objective, where the parameter ranges over levels of conformity. We present an optimization algorithm and establish its convergence to a stationary point. We show how to practically implement it using secure aggregation by interleaving iterations of the usual federated averaging method with device filtering. We conclude with numerical experiments on neural networks as well as linear models on tasks from computer vision and natural language processing.

References in corpus (34)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• EMNIST: an extension of MNIST to handwritten letters
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Fair Resource Allocation in Federated Learning
• UVeQFed: Universal Vector Quantization for Federated Learning
• Agnostic Federated Learning
• Over-the-Air Federated Learning from Heterogeneous Data
• Personalized Federated Learning with Moreau Envelopes
• Federated Learning: A Signal Processing Perspective
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Bayesian Nonparametric Federated Learning of Neural Networks
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization
• Minimax Statistical Learning with Wasserstein Distances
• Exploiting Shared Representations for Personalized Federated Learning
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Large-Scale Methods for Distributionally Robust Optimization
• The Skellam Mechanism for Differentially Private Federated Learning
• Federated Learning with Partial Model Personalization
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Variance-based regularization with convex objectives
• Adaptive Sampling for Stochastic Risk-Averse Learning
• Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
• A Smoother Way to Train Structured Prediction Models