Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
arXiv:2110.09246
Abstract
Detecting out-of-distribution (OOD) samples is vital for developing machine learning based models for critical safety systems. Common approaches for OOD detection assume access to some OOD samples during training which may not be available in a real-life scenario. Instead, we utilize the {\em predictive normalized maximum likelihood} (pNML) learner, in which no assumptions are made on the tested input. We derive an explicit expression of the pNML and its generalization error, denoted as the {\em regret}, for a single layer neural network (NN). We show that this learner generalizes well when (i) the test vector resides in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, or (ii) the test sample is far from the decision boundary. Furthermore, we describe how to efficiently apply the derived pNML regret to any pretrained deep NN, by employing the explicit pNML for the last layer, followed by the softmax function. Applying the derived regret to deep NN requires neither additional tunable parameters nor extra data. We extensively evaluate our approach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and WideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30 showing a significant improvement of up to 15.6\% over recent leading methods.
arXiv commentNeurIPS 2021
References in corpus (15)
- Wide Residual Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Reconciling modern machine learning practice and the bias-variance trade-off
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Energy-based Out-of-distribution Detection
- Predictive Uncertainty Estimation via Prior Networks
- TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
- Spectrally-normalized margin bounds for neural networks
- Recovery Guarantees for One-hidden-layer Neural Networks
- A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
- Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks
- Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation
- Balancing Specialization, Generalization, and Compression for Detection and Tracking
- Utilizing Adversarial Targeted Attacks to Boost Adversarial Robustness