Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation

Abstract

Fully Test-Time Adaptation (FTTA) addresses domain shifts without access to source data and training protocols of the pre-trained models. Traditional strategies that align source and target feature distributions are infeasible in FTTA due to the absence of training data and unpredictable target domains. In this work, we exploit a dual perspective on FTTA, and propose Agnostic FTTA (AFTTA) as a novel formulation that enables the usage of off-the-shelf domain transformations during test-time to enable direct generalization to unforeseeable target data. To address this, we develop an uncover-and-unlearn approach. First, we uncover potential unwanted shifts between source and target domains by simulating them through predefined mappings and consider them as nuisances. Then, during test-time prediction, the model is enforced to unlearn these nuisances by regularizing the consequent shifts in latent representations and label predictions. Specifically, a mutual information-based criterion is devised and applied to guide nuisances unlearning in the feature space and encourage confident and consistent prediction in label space. Our proposed approach explicitly addresses agnostic domain shifts, enabling superior model generalization under FTTA constraints. Extensive experiments on various tasks, involving corruption and style shifts, demonstrate that our method consistently outperforms existing approaches.

26 pages, 4 figures

References in corpus (16)

• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Model Adaptation: Unsupervised Domain Adaptation without Source Data
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Universal Domain Adaptation through Self Supervision
• Improving robustness against common corruptions by covariate shift adaptation
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Efficient Test-Time Model Adaptation without Forgetting
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation
• Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing
• Encoding Robustness to Image Style via Adversarial Feature Perturbations