When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights

Abstract

As a crucial step toward real-world learning scenarios with changing environments, dataset shift theory and invariant representation learning algorithm have been extensively studied to relax the identical distribution assumption in classical learning setting. Among the different assumptions on the essential of shifting distributions, generalized label shift (GLS) is the latest developed one which shows great potential to deal with the complex factors within the shift. In this paper, we aim to explore the limitations of current dataset shift theory and algorithm, and further provide new insights by presenting a comprehensive understanding of GLS. From theoretical aspect, two informative generalization bounds are derived, and the GLS learner is proved to be sufficiently close to optimal target model from the Bayesian perspective. The main results show the insufficiency of invariant representation learning, and prove the sufficiency and necessity of GLS correction for generalization, which provide theoretical supports and innovations for exploring generalizable model under dataset shift. From methodological aspect, we provide a unified view of existing shift correction frameworks, and propose a kernel embedding-based correction algorithm (KECA) to minimize the generalization error and achieve successful knowledge transfer. Both theoretical results and extensive experiment evaluations demonstrate the sufficiency and necessity of GLS correction for addressing dataset shift and the superiority of proposed algorithm.

arXiv commentAccepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (15)

• Learning Transferable Features with Deep Adaptation Networks
• Conditional Adversarial Domain Adaptation
• VisDA: The Visual Domain Adaptation Challenge
• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
• Bridging Theory and Algorithm for Domain Adaptation
• Unsupervised Domain Adaptation via Discriminative Manifold Propagation
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
• Cross-Site Severity Assessment of COVID-19 from CT Images via Domain Adaptation
• Conditional Generative Moment-Matching Networks
• A Rigorous Theory of Conditional Mean Embeddings
• A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Fundamental Limits and Tradeoffs in Invariant Representation Learning