Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection

Abstract

While many real-world data streams imply that they change frequently in a nonstationary way, most of deep learning methods optimize neural networks on training data, and this leads to severe performance degradation when dataset shift happens. However, it is less possible to annotate or inspect newly streamed data by humans, and thus it is desired to measure model drift at inference time in an unsupervised manner. In this paper, we propose a novel method of model drift estimation by exploiting statistics of batch normalization layer on unlabeled test data. To remedy possible sampling error of streamed input data, we adopt low-rank approximation to each representational layer. We show the effectiveness of our method not only on dataset shift detection but also on model selection when there are multiple candidate models among model zoo or training trajectories in an unsupervised way. We further demonstrate the consistency of our method by comparing model drift scores between different network architectures.

11 pages, 5 figures, 2 tables

References in corpus (5)

• Large Batch Training of Convolutional Networks
• Improving model calibration with accuracy versus uncertainty optimization
• Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics
• Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation
• A linearized framework and a new benchmark for model selection for fine-tuning