Model-agnostic out-of-distribution detection using combined statistical tests

Abstract

We present simple methods for out-of-distribution detection using a trained generative model. These techniques, based on classical statistical tests, are model-agnostic in the sense that they can be applied to any differentiable generative model. The idea is to combine a classical parametric test (Rao's score test) with the recently introduced typicality test. These two test statistics are both theoretically well-founded and exploit different sources of information based on the likelihood for the typicality test and its gradient for the score test. We show that combining them using Fisher's method overall leads to a more accurate out-of-distribution test. We also discuss the benefits of casting out-of-distribution detection as a statistical testing problem, noting in particular that false positive rate control can be valuable for practical out-of-distribution detection. Despite their simplicity and generality, these methods can be competitive with model-specific out-of-distribution detection algorithms without any assumptions on the out-distribution.

arXiv commentAccepted at the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Importance Weighted Autoencoders
• Learning Deep Kernels for Non-Parametric Two-Sample Tests
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features
• Efficient Per-Example Gradient Computations
• not-MIWAE: Deep Generative Modelling with Missing not at Random Data
• Robust Out-of-Distribution Detection on Deep Probabilistic Generative Models