A t-distribution based operator for enhancing out of distribution robustness of neural network classifiers

Abstract

Neural Network (NN) classifiers can assign extreme probabilities to samples that have not appeared during training (out-of-distribution samples) resulting in erroneous and unreliable predictions. One of the causes for this unwanted behaviour lies in the use of the standard softmax operator which pushes the posterior probabilities to be either zero or unity hence failing to model uncertainty. The statistical derivation of the softmax operator relies on the assumption that the distributions of the latent variables for a given class are Gaussian with known variance. However, it is possible to use different assumptions in the same derivation and attain from other families of distributions as well. This allows derivation of novel operators with more favourable properties. Here, a novel operator is proposed that is derived using $t$-distributions which are capable of providing a better description of uncertainty. It is shown that classifiers that adopt this novel operator can be more robust to out of distribution samples, often outperforming NNs that use the standard softmax operator. These enhancements can be reached with minimal changes to the NN architecture.

arXiv comment5 pages, 5 figures, to be published in IEEE Signal Processing Letters, reproducible code https://github.com/idiap/tsoftmax

References in corpus (11)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Additive Margin Softmax for Face Verification
• Deep Learning for Audio Signal Processing
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Machine learning in acoustics: theory and applications
• Deep Anomaly Detection with Outlier Exposure
• Learning Confidence for Out-of-Distribution Detection in Neural Networks