KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation

Abstract

Conventional unsupervised multi-source domain adaptation (UMDA) methods assume all source domains can be accessed directly. This neglects the privacy-preserving policy, that is, all the data and computations must be kept decentralized. There exists three problems in this scenario: (1) Minimizing the domain distance requires the pairwise calculation of the data from source and target domains, which is not accessible. (2) The communication cost and privacy security limit the application of UMDA methods (e.g., the domain adversarial training). (3) Since users have no authority to check the data quality, the irrelevant or malicious source domains are more likely to appear, which causes negative transfer. In this study, we propose a privacy-preserving UMDA paradigm named Knowledge Distillation based Decentralized Domain Adaptation (KD3A), which performs domain adaptation through the knowledge distillation on models from different source domains. KD3A solves the above problems with three components: (1) A multi-source knowledge distillation method named Knowledge Vote to learn high-quality domain consensus knowledge. (2) A dynamic weighting strategy named Consensus Focus to identify both the malicious and irrelevant domains. (3) A decentralized optimization strategy for domain distance named BatchNorm MMD. The extensive experiments on DomainNet demonstrate that KD3A is robust to the negative transfer and brings a 100x reduction of communication cost compared with other decentralized UMDA methods. Moreover, our KD3A significantly outperforms state-of-the-art UMDA approaches.

arXiv comment15 pages, 5 figures, Accepted for presentation at ICML2021

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• mixup: Beyond Empirical Risk Minimization
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Conditional Adversarial Domain Adaptation
• How To Backdoor Federated Learning
• Model Adaptation: Unsupervised Domain Adaptation without Source Data
• On Learning Invariant Representation for Domain Adaptation
• Federated Adversarial Domain Adaptation
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Knowledge Adaptation: Teaching to Adapt
• Electrocardio Panorama: Synthesizing New ECG Views with Self-supervision

Cited by in corpus (5)

• Uncertainty-Guided Mixup for Semi-Supervised Domain Adaptation without Source Data
• Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable Transfer from Black-Box to Lightweight Segmentation Model
• Domain Adaptation without Model Transferring
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Electrocardio Panorama: Synthesizing New ECG Views with Self-supervision