Revisiting Batch Normalization For Practical Domain Adaptation

Abstract

Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al. 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.

References in corpus (8)

• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Domain Separation Networks
• Rethinking the Inception Architecture for Computer Vision
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation

Cited by in corpus (98)

• Deep Visual Domain Adaptation: A Survey
• Applications of Unsupervised Deep Transfer Learning to Intelligent Fault Diagnosis: A Survey and Comparative Study
• Asymmetric Tri-training for Unsupervised Domain Adaptation
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Multimodal Unsupervised Image-to-Image Translation
• Tent: Fully Test-time Adaptation by Entropy Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Generative Adversarial Network for Medical Images (MI-GAN)
• Slimmable Neural Networks
• Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Universal Domain Adaptation through Self Supervision
• Self-supervised Domain Adaptation for Computer Vision Tasks
• Improving robustness against common corruptions by covariate shift adaptation
• Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras
• Federated Adversarial Domain Adaptation
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
• Unsupervised Domain Adaptation with Adversarial Residual Transform Networks
• Neural Style Transfer: A Review
• Integrating Expert Knowledge with Domain Adaptation for Unsupervised Fault Diagnosis
• Robust Unsupervised Domain Adaptation for Neural Networks via Moment Alignment
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Cross-regional oil palm tree counting and detection via multi-level attention domain adaptation network
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Demystifying Neural Style Transfer
• Few-Shot Unsupervised Image-to-Image Translation
• Semi-supervised Domain Adaptation via Minimax Entropy
• Missing-Class-Robust Domain Adaptation by Unilateral Alignment for Fault Diagnosis
• Salient Object Detection with Lossless Feature Reflection and Weighted Structural Loss
• TimeMatch: Unsupervised Cross-Region Adaptation by Temporal Shift Estimation
• Self-ensembling for visual domain adaptation
• Dual Adaptive Representation Alignment for Cross-domain Few-shot Learning
• A Survey on Deep Domain Adaptation for LiDAR Perception
• Label Efficient Learning of Transferable Representations across Domains and Tasks
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
• Object Detection Using Deep CNNs Trained on Synthetic Images
• Physically-Constrained Transfer Learning through Shared Abundance Space for Hyperspectral Image Classification
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• Inferring Latent Domains for Unsupervised Deep Domain Adaptation
• Feature Alignment and Restoration for Domain Generalization and Adaptation
• Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer
• Rethinking "Batch" in BatchNorm
• Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains
• FS-BAN: Born-Again Networks for Domain Generalization Few-Shot Classification
• Domain Adaptive Transfer Learning for Fault Diagnosis
• Test-time Batch Statistics Calibration for Covariate Shift
• Style Normalization and Restitution for Generalizable Person Re-identification
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Importance Weighted Adversarial Nets for Partial Domain Adaptation
• Temporal Attentive Alignment for Large-Scale Video Domain Adaptation
• Knowledge distillation via adaptive instance normalization
• MixNorm: Test-Time Adaptation Through Online Normalization Estimation
• Keep it Simple: Image Statistics Matching for Domain Adaptation
• Image Generation From Small Datasets via Batch Statistics Adaptation
• Correlation Alignment by Riemannian Metric for Domain Adaptation
• AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching
• Online Supervised Training of Spaceborne Vision during Proximity Operations using Adaptive Kalman Filtering
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• Domain Adaptation Through Task Distillation
• Online Adaptation through Meta-Learning for Stereo Depth Estimation
• Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation
• SF-UDA${}^{3D}$: Source-Free Unsupervised Domain Adaptation for LiDAR-Based 3D Object Detection
• Batch Group Normalization
• Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
• SCIDA: Self-Correction Integrated Domain Adaptation from Single- to Multi-label Aerial Images
• IDM: An Intermediate Domain Module for Domain Adaptive Person Re-ID
• Rethinking the Distribution Gap of Person Re-identification with Camera-based Batch Normalization
• Towards Defending Multiple ${\ell }_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization
• 3D-to-2D Distillation for Indoor Scene Parsing
• A Lifelong Learning Approach to Brain MR Segmentation Across Scanners and Protocols
• Does Data Augmentation Benefit from Split BatchNorms
• Learning to Transfer for Traffic Forecasting via Multi-task Learning
• Feature Stylization and Domain-aware Contrastive Learning for Domain Generalization
• Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers
• Predicting with High Correlation Features
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• Iris Style Transfer: Enhancing Iris Recognition with Style Features and Privacy Preservation through Neural Style Transfer
• Encoding Robustness to Image Style via Adversarial Feature Perturbations
• Unsupervised Domain Adaptation for Retinal Vessel Segmentation with Adversarial Learning and Transfer Normalization
• AdaFlow: Domain-Adaptive Density Estimator with Application to Anomaly Detection and Unpaired Cross-Domain Translation
• HyperFusion-Net: Densely Reflective Fusion for Salient Object Detection
• Style is a Distribution of Features
• Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net
• A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation
• Resolution Switchable Networks for Runtime Efficient Image Recognition
• Reducing the feature divergence of RGB and near-infrared images using Switchable Normalization
• Recognizing Instagram Filtered Images with Feature De-stylization
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• Towards Recognizing New Semantic Concepts in New Visual Domains
• Neural Random Projection: From the Initial Task To the Input Similarity Problem
• GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation
• Learning to see across Domains and Modalities
• Towards Data-Free Domain Generalization
• Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
• Towards Flops-constrained Face Recognition
• Double Forward Propagation for Memorized Batch Normalization
• WeightAlign: Normalizing Activations by Weight Alignment