FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation

Abstract

Federated adversary domain adaptation is a unique distributed minimax training task due to the prevalence of label imbalance among clients, with each client only seeing a subset of the classes of labels required to train a global model. To tackle this problem, we propose a distributed minimax optimizer referred to as FedMM, designed specifically for the federated adversary domain adaptation problem. It works well even in the extreme case where each client has different label classes and some clients only have unsupervised tasks. We prove that FedMM ensures convergence to a stationary point with domain-shifted unsupervised data. On a variety of benchmark datasets, extensive experiments show that FedMM consistently achieves either significant communication savings or significant accuracy improvements over federated optimizers based on the gradient descent ascent (GDA) algorithm. When training from scratch, for example, it outperforms other GDA based federated average methods by around $20\%$ in accuracy over the same communication rounds; and it consistently outperforms when training from pre-trained models with an accuracy improvement from $5.4\%$ to $9\%$ for different networks.

arXiv comment34 pages

References in corpus (15)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Conditional Adversarial Domain Adaptation
• Federated Optimization in Heterogeneous Networks
• Bridging Theory and Algorithm for Domain Adaptation
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• Federated Adversarial Domain Adaptation
• Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applications
• Federated Learning Based on Dynamic Regularization
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Breaking the Communication-Privacy-Accuracy Trilemma
• FedGAN: Federated Generative Adversarial Networks for Distributed Data
• Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency