Multiple Source Adaptation and the Renyi Divergence

Abstract

This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (1)

• Domain Adaptation for Statistical Classifiers

Cited by in corpus (24)

• Rényi Divergence and Kullback-Leibler Divergence
• Renyi Differential Privacy
• A review of domain adaptation without target labels
• Domain-Adversarial Neural Networks
• Bridging Theory and Algorithm for Domain Adaptation
• Federated Learning: A Signal Processing Perspective
• On Learning Invariant Representation for Domain Adaptation
• Domain Aggregation Networks for Multi-Source Domain Adaptation
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation
• A survey on domain adaptation theory: learning bounds and theoretical guarantees
• PAC-Bayes and Domain Adaptation
• Aggregating From Multiple Target-Shifted Sources
• A Discriminative Technique for Multiple-Source Adaptation
• Learn to Expect the Unexpected: Probably Approximately Correct Domain Generalization
• A Theory of Multiple-Source Adaptation with Limited Target Labeled Data
• Empirically Estimable Classification Bounds Based on a New Divergence Measure
• PAC-Bayesian Theorems for Domain Adaptation with Specialization to Linear Classifiers
• Moment-Based Domain Adaptation: Learning Bounds and Algorithms
• Multiple-Source Adaptation for Regression Problems
• Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm
• Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation
• A Computationally Efficient Classification Algorithm in Posterior Drift Model: Phase Transition and Minimax Adaptivity
• Entropies and their Asymptotic Theory in the discrete case
• Practical Estimation of Renyi Entropy