Robust Federated Learning: The Case of Affine Distribution Shifts

Abstract

Federated learning is a distributed paradigm that aims at training models using samples distributed across multiple users in a network while keeping the samples on users' devices with the aim of efficiency and protecting users privacy. In such settings, the training data is often statistically heterogeneous and manifests various distribution shifts across users, which degrades the performance of the learnt model. The primary goal of this paper is to develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples. To achieve this goal, we first consider a structured affine distribution shift in users' data that captures the device-dependent data heterogeneity in federated settings. This perturbation model is applicable to various federated learning problems such as image classification where the images undergo device-dependent imperfections, e.g. different intensity, contrast, and brightness. To address affine distribution shifts across users, we propose a Federated Learning framework Robust to Affine distribution shifts (FLRA) that is provably robust against affine Wasserstein shifts to the distribution of observed samples. To solve the FLRA's distributed minimax problem, we propose a fast and efficient optimization method and provide convergence guarantees via a gradient Descent Ascent (GDA) method. We further prove generalization error bounds for the learnt classifier to show proper generalization from empirical distribution of samples to the true underlying distribution. We perform several numerical experiments to empirically support FLRA. We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user, and our proposed algorithm achieves a significant gain in comparison to standard federated learning and adversarial training methods.

References in corpus (19)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Explaining and Harnessing Adversarial Examples
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Differentially Private Federated Learning: A Client Level Perspective
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Local SGD Converges Fast and Communicates Little
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• On the Convergence of Local Descent Methods in Federated Learning
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
• Regularization via Mass Transportation
• Differentially Private Learning with Adaptive Clipping
• Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems
• Differentially Private Meta-Learning

Cited by in corpus (9)

• An Optimal Transport Approach to Personalized Federated Learning
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Auxo: Efficient Federated Learning via Scalable Client Clustering
• Mitigating Group Bias in Federated Learning for Heterogeneous Devices
• CDMA: A Practical Cross-Device Federated Learning Algorithm for General Minimax Problems
• FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation
• A Wasserstein Minimax Framework for Mixed Linear Regression
• Compositional federated learning: Applications in distributionally robust averaging and meta learning