Test-time Batch Statistics Calibration for Covariate Shift

Abstract

Deep neural networks have a clear degradation when applying to the unseen environment due to the covariate shift. Conventional approaches like domain adaptation requires the pre-collected target data for iterative training, which is impractical in real-world applications. In this paper, we propose to adapt the deep models to the novel environment during inference. An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics. However, we show that test time normalization may potentially deteriorate the discriminative structures due to the mismatch between target batch statistics and source parameters. To this end, we present a general formulation $\alpha$-BN to calibrate the batch statistics by mixing up the source and target statistics for both alleviating the domain shift and preserving the discriminative structures. Based on $\alpha$-BN, we further present a novel loss function to form a unified test time adaptation framework Core, which performs the pairwise class correlation online optimization. Extensive experiments show that our approaches achieve the state-of-the-art performance on total twelve datasets from three topics, including model robustness to corruptions, domain generalization on image classification and semantic segmentation. Particularly, our $\alpha$-BN improves 28.4\% to 43.9\% on GTA5 $\to$ Cityscapes without any training, even outperforms the latest source-free domain adaptation method.

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Learning Transferable Features with Deep Adaptation Networks
• Wide Residual Networks
• Conditional Adversarial Domain Adaptation
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification
• Revisiting Batch Normalization For Practical Domain Adaptation
• Maximum Density Divergence for Domain Adaptation
• Tent: Fully Test-time Adaptation by Entropy Minimization
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
• RobustBench: a standardized adversarial robustness benchmark
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• In Search of Lost Domain Generalization
• Self-Challenging Improves Cross-Domain Generalization
• Domain Generalization with MixStyle

Cited by in corpus (2)

• SATS: Self-Attention Transfer for Continual Semantic Segmentation
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation