Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features

Abstract

Deep generative networks trained via maximum likelihood on a natural image dataset like CIFAR10 often assign high likelihoods to images from datasets with different objects (e.g., SVHN). We refine previous investigations of this failure at anomaly detection for invertible generative networks and provide a clear explanation of it as a combination of model bias and domain prior: Convolutional networks learn similar low-level feature distributions when trained on any natural image dataset and these low-level features dominate the likelihood. Hence, when the discriminative features between inliers and outliers are on a high-level, e.g., object shapes, anomaly detection becomes particularly challenging. To remove the negative impact of model bias and domain prior on detecting high-level differences, we propose two methods, first, using the log likelihood ratios of two identical models, one trained on the in-distribution data (e.g., CIFAR10) and the other one on a more general distribution of images (e.g., 80 Million Tiny Images). We also derive a novel outlier loss for the in-distribution network on samples from the more general distribution to further improve the performance. Secondly, using a multi-scale model like Glow, we show that low-level features are mainly captured at early scales. Therefore, using only the likelihood contribution of the final scale performs remarkably well for detecting high-level feature differences of the out-of-distribution and the in-distribution. This method is especially useful if one does not have access to a suitable general distribution. Overall, our methods achieve strong anomaly detection performance in the unsupervised setting, and only slightly underperform state-of-the-art classifier-based methods in the supervised setting. Code can be found at https://github.com/boschresearch/hierarchical_anomaly_detection.

arXiv commentPublished at NeurIPS 2020. Code can be found at https://github.com/boschresearch/hierarchical_anomaly_detection

References in corpus (17)

• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Density estimation using Real NVP
• Deep Anomaly Detection with Outlier Exposure
• Deep Anomaly Detection Using Geometric Transformations
• Input complexity and out-of-distribution detection with likelihood-based generative models
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Classification-Based Anomaly Detection for General Data
• OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations
• Contrastive Training for Improved Out-of-Distribution Detection
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Deep Semi-Supervised Anomaly Detection
• Rethinking Assumptions in Deep Anomaly Detection
• Understanding the (un)interpretability of natural image distributions using generative models
• Out-of-Distribution Detection Using Neural Rendering Generative Models

Cited by in corpus (11)

• On the Importance of Gradients for Detecting Distributional Shifts in the Wild
• Perfect density models cannot guarantee anomaly detection
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Hierarchical VAEs Know What They Don't Know
• SLA${}^2$P: Self-supervised Anomaly Detection with Adversarial Perturbation
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• Unsupervised Anomaly Detection From Semantic Similarity Scores
• Model-agnostic out-of-distribution detection using combined statistical tests
• On Out-of-distribution Detection with Energy-based Models
• New Methods and Datasets for Group Anomaly Detection From Fundamental Physics