Input complexity and out-of-distribution detection with likelihood-based generative models

Abstract

Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

Accepted for ICLR2020

References in corpus (6)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Deep Anomaly Detection with Outlier Exposure
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Uncertainty in the Variational Information Bottleneck

Cited by in corpus (21)

• A Unifying Review of Deep and Shallow Anomaly Detection
• Generalized Out-of-Distribution Detection: A Survey
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• The relationship between fine galaxy stellar morphology and star formation activity in cosmological simulations: a deep learning view
• Can multi-label classification networks know what they don't know?
• Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features
• Robust Out-of-distribution Detection for Neural Networks
• Perfect density models cannot guarantee anomaly detection
• Density of States Estimation for Out-of-Distribution Detection
• TRUST-LAPSE: An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring
• SoK: Machine Learning Governance
• Hybrid Open-set Segmentation with Synthetic Negative Data
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders
• Hybrid Models for Open Set Recognition
• Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• A versatile anomaly detection method for medical images with a flow-based generative model in semi-supervision setting
• DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
• Unsupervised Class-Incremental Learning Through Confusion
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models