Uncertainty-Aware Reliable Text Classification

Abstract

Deep neural networks have significantly contributed to the success in predictive accuracy for classification tasks. However, they tend to make over-confident predictions in real-world settings, where domain shifting and out-of-distribution (OOD) examples exist. Most research on uncertainty estimation focuses on computer vision because it provides visual validation on uncertainty quality. However, few have been presented in the natural language process domain. Unlike Bayesian methods that indirectly infer uncertainty through weight uncertainties, current evidential uncertainty-based methods explicitly model the uncertainty of class probabilities through subjective opinions. They further consider inherent uncertainty in data with different root causes, vacuity (i.e., uncertainty due to a lack of evidence) and dissonance (i.e., uncertainty due to conflicting evidence). In our paper, we firstly apply evidential uncertainty in OOD detection for text classification tasks. We propose an inexpensive framework that adopts both auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class, which has high vacuity for OOD samples. Extensive empirical experiments demonstrate that our model based on evidential uncertainty outperforms other counterparts for detecting OOD examples. Our approach can be easily deployed to traditional recurrent neural networks and fine-tuned pre-trained transformers.

arXiv commentKDD 2021

References in corpus (19)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• mixup: Beyond Empirical Risk Minimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Weight Uncertainty in Neural Networks
• Evidential Deep Learning to Quantify Classification Uncertainty
• A large annotated corpus for learning natural language inference
• Deep Anomaly Detection with Outlier Exposure
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Out-of-distribution Detection in Classifiers via Generation
• Deep Active Learning for Named Entity Recognition
• Pretrained Transformers Improve Out-of-Distribution Robustness
• Quantifying Classification Uncertainty using Regularized Evidential Neural Networks
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
• Uncertainty-Aware Deep Classifiers using Generative Models