WAIC, but Why? Generative Ensembles for Robust Anomaly Detection

Abstract

Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Deep Anomaly Detection with Outlier Exposure
• Uncertainty in the Variational Information Bottleneck
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Novelty Detection with GAN

Cited by in corpus (75)

• Deep Learning for Anomaly Detection: A Survey
• Energy-based Out-of-distribution Detection
• Generalized Out-of-Distribution Detection: A Survey
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Input complexity and out-of-distribution detection with likelihood-based generative models
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Learning and Evaluating Representations for Deep One-class Classification
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Contrastive Training for Improved Out-of-Distribution Detection
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Unsupervised Brain Anomaly Detection and Segmentation with Transformers
• Probabilistic Autoencoder
• Monitoring and explainability of models in production
• Anomaly localization by modeling perceptual features
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• Perfect density models cannot guarantee anomaly detection
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data
• Density of States Estimation for Out-of-Distribution Detection
• Novelty Detection Via Blurring
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• TRUST-LAPSE: An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring
• Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems
• The Hidden Uncertainty in a Neural Networks Activations
• Boundary of Distribution Support Generator (BDSG): Sample Generation on the Boundary
• Fully probabilistic quasar continua predictions near Lyman-α with conditional neural spline flows
• A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders
• Further Analysis of Outlier Detection with Deep Generative Models
• Regularized Autoencoders via Relaxed Injective Probability Flow
• Anomalous Example Detection in Deep Learning: A Survey
• Likelihood Ratios for Out-of-Distribution Detection
• Hierarchical VAEs Know What They Don't Know
• Neural Density Estimation and Likelihood-free Inference
• Out-of-Distribution Detection Using Neural Rendering Generative Models
• Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• InFlow: Robust outlier detection utilizing Normalizing Flows
• Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection
• Bounds all around: training energy-based models with bidirectional bounds
• Flow-based Self-supervised Density Estimation for Anomalous Sound Detection
• Regularizing Attention Networks for Anomaly Detection in Visual Question Answering
• Unsupervised anomaly localization in high-resolution breast scans using deep pluralistic image completion
• Sliced Iterative Normalizing Flows
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Deep Generative Models Strike Back! Improving Understanding and Evaluation in Light of Unmet Expectations for OoD Data
• Bigeminal Priors Variational auto-encoder
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• Detecting Anomalous Event Sequences with Temporal Point Processes
• An Algorithm for Out-Of-Distribution Attack to Neural Network Encoder
• Rectangular Flows for Manifold Learning
• Likelihood Assignment for Out-of-Distribution Inputs in Deep Generative Models is Sensitive to Prior Distribution Choice
• On Out-of-distribution Detection with Energy-based Models
• Reliable uncertainty estimate for antibiotic resistance classification with Stochastic Gradient Langevin Dynamics
• Towards Robust Active Feature Acquisition
• DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
• Detecting and Understanding Generalization Barriers for Neural Machine Translation
• Probabilistic Trust Intervals for Out of Distribution Detection
• Conditional Invertible Neural Networks for Diverse Image-to-Image Translation
• Natural Attribute-based Shift Detection
• Reject Illegal Inputs with Generative Classifier Derived from Any Discriminative Classifier
• Unsupervised Learning of Multi-level Structures for Anomaly Detection
• Multi-Class Anomaly Detection
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Entropic Issues in Likelihood-Based OOD Detection
• Test for non-negligible adverse shifts