Hierarchical VAEs Know What They Don't Know
arXiv:2102.08248
Abstract
Deep generative models have been demonstrated as state-of-the-art density estimators. Yet, recent work has found that they often assign a higher likelihood to data from outside the training distribution. This seemingly paradoxical behavior has caused concerns over the quality of the attained density estimates. In the context of hierarchical variational autoencoders, we provide evidence to explain this behavior by out-of-distribution data having in-distribution low-level features. We argue that this is both expected and desirable behavior. With this insight in hand, we develop a fast, scalable and fully unsupervised likelihood-ratio score for OOD detection that requires data to be in-distribution across all feature-levels. We benchmark the method on a vast set of data and model combinations and achieve state-of-the-art results on out-of-distribution detection.
arXiv commentAppeared in Proceedings of the 38th International Conference on Machine Learning (ICML 2021). 18 pages, source code available at https://github.com/JakobHavtorn/hvae-oodd, https://github.com/vlievin/biva-pytorch and https://github.com/larsmaaloee/BIVA
References in corpus (19)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- WaveNet: A Generative Model for Raw Audio
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Pixel Recurrent Neural Networks
- Variational Inference with Normalizing Flows
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Deep Anomaly Detection with Outlier Exposure
- Auxiliary Deep Generative Models
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Inference Suboptimality in Variational Autoencoders
- Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
- Uncertainty in the Variational Information Bottleneck
- Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- Avoiding Latent Variable Collapse With Generative Skip Models
- Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features