Entropic Issues in Likelihood-Based OOD Detection

Abstract

Deep generative models trained by maximum likelihood remain very popular methods for reasoning about data probabilistically. However, it has been observed that they can assign higher likelihoods to out-of-distribution (OOD) data than in-distribution data, thus calling into question the meaning of these likelihood values. In this work we provide a novel perspective on this phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour mentioned above, suppressing likelihood values on datasets with higher entropy. Although our idea is simple, we have not seen it explored yet in the literature. This analysis provides further explanation for the success of OOD detection methods based on likelihood ratios, as the problematic entropy term cancels out in expectation. Finally, we discuss how this observation relates to recent success in OOD detection with manifold-supported models, for which the above decomposition does not hold directly.

arXiv commentAccepted to the NeurIPS 2021 ICBINB Workshop. Extended the result in expectation to a result with high probability

References in corpus (13)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Normalizing Flows: An Introduction and Review of Current Methods
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Implicit Generation and Generalization in Energy-Based Models
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• A non-parametric k-nearest neighbour entropy estimator
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Perfect density models cannot guarantee anomaly detection
• Deep Generative Models Strike Back! Improving Understanding and Evaluation in Light of Unmet Expectations for OoD Data
• Rectangular Flows for Manifold Learning