Federated Learning with Superquantile Aggregation for Heterogeneous Data

Abstract

We present a federated learning framework that is designed to robustly deliver good predictive performance across individual clients with heterogeneous data. The proposed approach hinges upon a superquantile-based learning objective that captures the tail statistics of the error distribution over heterogeneous clients. We present a stochastic training algorithm that interleaves differentially private client filtering with federated averaging steps. We prove finite time convergence guarantees for the algorithm: $O(1/\sqrt{T})$ in the nonconvex case in $T$ communication rounds and $O(\exp (-T/{\kappa }^{3/2})+\kappa /T)$ in the strongly convex case with local condition number $\kappa$. Experimental results on benchmark datasets for federated learning demonstrate that our approach is competitive with classical ones in terms of average error and outperforms them in terms of tail statistics of the error.

arXiv commentMachine Learning Journal, Special Issue on Safe and Fair Machine Learning (To appear)

References in corpus (34)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• EMNIST: an extension of MNIST to handwritten letters
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Fair Resource Allocation in Federated Learning
• UVeQFed: Universal Vector Quantization for Federated Learning
• Agnostic Federated Learning
• Over-the-Air Federated Learning from Heterogeneous Data
• Personalized Federated Learning with Moreau Envelopes
• Federated Learning: A Signal Processing Perspective
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Bayesian Nonparametric Federated Learning of Neural Networks
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization
• Minimax Statistical Learning with Wasserstein Distances
• Exploiting Shared Representations for Personalized Federated Learning
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Large-Scale Methods for Distributionally Robust Optimization
• The Skellam Mechanism for Differentially Private Federated Learning
• Federated Learning with Partial Model Personalization
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Variance-based regularization with convex objectives
• Adaptive Sampling for Stochastic Risk-Averse Learning
• Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
• A Smoother Way to Train Structured Prediction Models