Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples

Abstract

In this paper, we argue that the unsatisfactory out-of-distribution (OOD) detection performance of neural networks is mainly due to the SoftMax loss anisotropy and propensity to produce low entropy probability distributions in disagreement with the principle of maximum entropy. Current out-of-distribution (OOD) detection approaches usually do not directly fix the SoftMax loss drawbacks, but rather build techniques to circumvent it. Unfortunately, those methods usually produce undesired side effects (e.g., classification accuracy drop, additional hyperparameters, slower inferences, and collecting extra data). In the opposite direction, we propose replacing SoftMax loss with a novel loss function that does not suffer from the mentioned weaknesses. The proposed IsoMax loss is isotropic (exclusively distance-based) and provides high entropy posterior probability distributions. Replacing the SoftMax loss by IsoMax loss requires no model or training changes. Additionally, the models trained with IsoMax loss produce as fast and energy-efficient inferences as those trained using SoftMax loss. Moreover, no classification accuracy drop is observed. The proposed method does not rely on outlier/background data, hyperparameter tuning, temperature calibration, feature extraction, metric learning, adversarial training, ensemble procedures, or generative models. Our experiments showed that IsoMax loss works as a seamless SoftMax loss drop-in replacement that significantly improves neural networks' OOD detection performance. Hence, it may be used as a baseline OOD detection approach to be combined with current or future OOD detection techniques to achieve even higher results.

arXiv commentAccepted for publication in the IEEE Transactions on Neural Networks and Learning Systems: Special Issue on Deep Learning for Anomaly Detection

References in corpus (30)

• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Prototypical Networks for Few-shot Learning
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Deep Learning for Anomaly Detection: A Review
• Deep Learning Face Representation by Joint Identification-Verification
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Additive Margin Softmax for Face Verification
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• Towards Open World Recognition
• Energy-based Out-of-distribution Detection
• Deep Anomaly Detection with Outlier Exposure
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Predictive Uncertainty Estimation via Prior Networks
• The Extreme Value Machine
• The Computational Limits of Deep Learning
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Adversarial Training Can Hurt Generalization
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• Reducing Network Agnostophobia
• Maximum-Entropy Fine-Grained Classification
• Confidence estimation in Deep Neural networks via density modelling
• Entropic Out-of-Distribution Detection
• Soft Labeling Affects Out-of-Distribution Detection of Deep Neural Networks