Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models

Abstract

Recent research has revealed that deep generative models including flow-based models and Variational Autoencoders may assign higher likelihoods to out-of-distribution (OOD) data than in-distribution (ID) data. However, we cannot sample OOD data from the model. This counterintuitive phenomenon has not been satisfactorily explained and brings obstacles to OOD detection with flow-based models. In this paper, we prove theorems to investigate the Kullback-Leibler divergence in flow-based model and give two explanations for the above phenomenon. Based on our theoretical analysis, we propose a new method \PADmethod\ to leverage KL divergence and local pixel dependence of representations to perform anomaly detection. Experimental results on prevalent benchmarks demonstrate the effectiveness and robustness of our method. For group anomaly detection, our method achieves 98.1\% AUROC on average with a small batch size of 5. On the contrary, the baseline typicality test-based method only achieves 64.6\% AUROC on average due to its failure on challenging problems. Our method also outperforms the state-of-the-art method by 9.1\% AUROC. For point-wise anomaly detection, our method achieves 90.7\% AUROC on average and outperforms the baseline by 5.2\% AUROC. Besides, our method has the least notable failures and is the most robust one.

References in corpus (23)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Conditional Image Generation with PixelCNN Decoders
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• NICE: Non-linear Independent Components Estimation
• Deep Learning for Anomaly Detection: A Survey
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Deep Anomaly Detection with Outlier Exposure
• Normalizing Flows for Probabilistic Modeling and Inference
• Towards a Definition of Disentangled Representations
• Understanding disentangling in $\beta$-VAE
• An elementary introduction to information geometry
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Input complexity and out-of-distribution detection with likelihood-based generative models
• Ensemble estimation of multivariate f-divergence
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Are generative deep models for novelty detection truly better?
• Semi-Conditional Normalizing Flows for Semi-Supervised Learning
• Transflow Learning: Repurposing Flow Models Without Retraining