DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models

Abstract

To ensure robust and reliable classification results, OoD (out-of-distribution) indicators based on deep generative models are proposed recently and are shown to work well on small datasets. In this paper, we conduct the first large collection of benchmarks (containing 92 dataset pairs, which is 1 order of magnitude larger than previous ones) for existing OoD indicators and observe that none perform well. We thus advocate that a large collection of benchmarks is mandatory for evaluating OoD indicators. We propose a novel theoretical framework, DOI, for divergence-based Out-of-Distribution indicators (instead of traditional likelihood-based) in deep generative models. Following this framework, we further propose a simple and effective OoD detection algorithm: Single-shot Fine-tune. It significantly outperforms past works by 5~8 in AUROC, and its performance is close to optimal. In recent, the likelihood criterion is shown to be ineffective in detecting OoD. Single-shot Fine-tune proposes a novel fine-tune criterion to detect OoD, by whether the likelihood of the testing sample is improved after fine-tuning a well-trained model on it. Fine-tune criterion is a clear and easy-following criterion, which will lead the OoD domain into a new stage.

References in corpus (12)

• Conditional Image Generation with PixelCNN Decoders
• Wide Residual Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Density estimation using Real NVP
• Deep Anomaly Detection with Outlier Exposure
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Input complexity and out-of-distribution detection with likelihood-based generative models
• Maximum Entropy Generators for Energy-Based Models
• Uncertainty in the Variational Information Bottleneck
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Unsupervised Out-of-Distribution Detection with Batch Normalization