Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

Abstract

Covariate shift has been shown to sharply degrade both predictive accuracy and the calibration of uncertainty estimates for deep learning models. This is worrying, because covariate shift is prevalent in a wide range of real world deployment settings. However, in this paper, we note that frequently there exists the potential to access small unlabeled batches of the shifted data just before prediction time. This interesting observation enables a simple but surprisingly effective method which we call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. Using this one line code change, we achieve state-of-the-art on recent covariate shift benchmarks and an mCE of 60.28\% on the challenging ImageNet-C dataset; to our knowledge, this is the best result for any model that does not incorporate additional data augmentation or modification of the training pipeline. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness (e.g. deep ensembles) and combining the two further improves performance. Our findings are supported by detailed measurements of the effect of this strategy on model behavior across rigorous ablations on various dataset modalities. However, the method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift, and is therefore worthy of additional study. We include links to the data in our figures to improve reproducibility, including a Python notebooks that can be run to easily modify our analysis at https://colab.research.google.com/drive/11N0wDZnMQQuLrRwRoumDCrhSaIhkqjof.

References in corpus (25)

• Distilling the Knowledge in a Neural Network
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Learning Transferable Features with Deep Adaptation Networks
• Densely Connected Convolutional Networks
• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• A Learned Representation For Artistic Style
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Do ImageNet Classifiers Generalize to ImageNet?
• Revisiting Batch Normalization For Practical Domain Adaptation
• Modulating early visual processing by language
• Self-training with Noisy Student improves ImageNet classification
• On Empirical Comparisons of Optimizers for Deep Learning
• Natural Adversarial Examples
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Fixup Initialization: Residual Learning Without Normalization
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Support and Invertibility in Domain-Invariant Representations
• Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Twin-GAN -- Unpaired Cross-Domain Image Translation with Weight-Sharing GANs

Cited by in corpus (27)

• Tent: Fully Test-time Adaptation by Entropy Minimization
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• Improving robustness against common corruptions by covariate shift adaptation
• Review: Deep Learning in Electron Microscopy
• MEMO: Test Time Robustness via Adaptation and Augmentation
• UPL-SFDA: Uncertainty-aware Pseudo Label Guided Source-Free Domain Adaptation for Medical Image Segmentation
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation
• Rethinking "Batch" in BatchNorm
• Test-time Batch Statistics Calibration for Covariate Shift
• SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation
• Distill-SODA: Distilling Self-Supervised Vision Transformer for Source-Free Open-Set Domain Adaptation in Computational Pathology
• Assessing out-of-domain generalization for robust building damage detection
• Test time Adaptation through Perturbation Robustness
• Adaptive Denoising via GainTuning
• PIG: Prompt Images Guidance for Night-Time Scene Parsing
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• Information Geometrically Generalized Covariate Shift Adaptation
• DIRA: Dynamic Domain Incremental Regularised Adaptation
• Neurodevelopmental Age Estimation of Infants Using a 3D-Convolutional Neural Network Model based on Fusion MRI Sequences
• Test-Time Adaptation with Principal Component Analysis
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
• The Robust Semantic Segmentation UNCV2023 Challenge Results
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation
• Frustratingly Easy Uncertainty Estimation for Distribution Shift
• Covariate Shift in High-Dimensional Random Feature Regression