Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift
arXiv:2111.10344
Abstract
Covariate shifts are a common problem in predictive modeling on real-world problems. This paper proposes addressing the covariate shift problem by minimizing Maximum Mean Discrepancy (MMD) statistics between the training and test sets in either feature input space, feature representation space, or both. We designed three techniques that we call MMD Representation, MMD Mask, and MMD Hybrid to deal with the scenarios where only a distribution shift exists, only a missingness shift exists, or both types of shift exist, respectively. We find that integrating an MMD loss component helps models use the best features for generalization and avoid dangerous extrapolation as much as possible for each test sample. Models treated with this MMD approach show better performance, calibration, and extrapolation on the test set.
arXiv commenta short version accepted by NeurIPS DistShift Workshop 2021
References in corpus (8)
- Categorical Reparameterization with Gumbel-Softmax
- Learning Transferable Features with Deep Adaptation Networks
- Deep Domain Confusion: Maximizing for Domain Invariance
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- Generative Moment Matching Networks
- Training generative neural networks via Maximum Mean Discrepancy optimization
- Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
- A Linear-Time Kernel Goodness-of-Fit Test