Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection

Abstract

Despite their successes, deep neural networks may make unreliable predictions when faced with test data drawn from a distribution different to that of the training data, constituting a major problem for AI safety. While this has recently motivated the development of methods to detect such out-of-distribution (OoD) inputs, a robust solution is still lacking. We propose a new probabilistic, unsupervised approach to this problem based on a Bayesian variational autoencoder model, which estimates a full posterior distribution over the decoder parameters using stochastic gradient Markov chain Monte Carlo, instead of fitting a point estimate. We describe how information-theoretic measures based on this posterior can then be used to detect OoD inputs both in input space and in the model's latent space. We empirically demonstrate the effectiveness of our proposed approach.

arXiv comment21 pages, extended version with supplementary material

References in corpus (41)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Conditional Image Generation with PixelCNN Decoders
• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• A Conceptual Introduction to Hamiltonian Monte Carlo
• Efficient GAN-Based Anomaly Detection
• Bayesian Active Learning for Classification and Preference Learning
• Neural Architecture Optimization
• Stochastic Gradient Hamiltonian Monte Carlo
• Deep Anomaly Detection with Outlier Exposure
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Variational Continual Learning
• Syntax-Directed Variational Autoencoder for Structured Data
• Effective Sample Size for Importance Sampling based on discrepancy measures
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Practical Deep Learning with Bayesian Principles
• Inference Suboptimality in Variational Autoencoders
• Bayesian Optimization of Combinatorial Structures
• Amortized Inference Regularization
• Uncertainty in the Variational Information Bottleneck
• VAE with a VampPrior
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Constrained Bayesian Optimization for Automatic Chemical Design
• Conditioning by adaptive sampling for robust design
• Bayesian GAN
• All SMILES Variational Autoencoder
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• Tighter Variational Bounds are Not Necessarily Better
• Semi-Implicit Variational Inference
• A Less Biased Evaluation of Out-of-distribution Sample Detectors
• Building robust classifiers through generation of confident out of distribution examples
• Icebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian Model
• A COLD Approach to Generating Optimal Samples
• Bayesian Optimization with Approximate Set Kernels

Cited by in corpus (10)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• CVAD: A generic medical anomaly detector based on Cascade VAE