Transferable Calibration with Lower Bias and Variance in Domain Adaptation

Abstract

Domain Adaptation (DA) enables transferring a learning machine from a labeled source domain to an unlabeled target one. While remarkable advances have been made, most of the existing DA methods focus on improving the target accuracy at inference. How to estimate the predictive uncertainty of DA models is vital for decision-making in safety-critical scenarios but remains the boundary to explore. In this paper, we delve into the open problem of Calibration in DA, which is extremely challenging due to the coexistence of domain shift and the lack of target labels. We first reveal the dilemma that DA models learn higher accuracy at the expense of well-calibrated probabilities. Driven by this finding, we propose Transferable Calibration (TransCal) to achieve more accurate calibration with lower bias and variance in a unified hyperparameter-free optimization framework. As a general post-hoc calibration method, TransCal can be easily applied to recalibrate existing DA methods. Its efficacy has been justified both theoretically and empirically.

References in corpus (17)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• On Calibration of Modern Neural Networks
• Conditional Adversarial Domain Adaptation
• Weight Uncertainty in Neural Networks
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• VisDA: The Visual Domain Adaptation Challenge
• Bridging Theory and Algorithm for Domain Adaptation
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• On Learning Invariant Representation for Domain Adaptation
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
• Regularized Learning for Domain Adaptation under Label Shifts
• Maximum Likelihood with Bias-Corrected Calibration is Hard-To-Beat at Label Shift Adaptation

Cited by in corpus (3)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Cross-regional oil palm tree counting and detection via multi-level attention domain adaptation network
• Overcoming Failures of Imagination in AI Infused System Development and Deployment