Uncertainty Estimation in Autoregressive Structured Prediction

Abstract

Uncertainty estimation is important for ensuring safety and robustness of AI systems. While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Thus, this work aims to investigate uncertainty estimation for autoregressive structured prediction tasks within a single unified and interpretable probabilistic ensemble-based framework. We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty; and discuss both the theoretical and practical challenges associated with obtaining them. This work also provides baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT'14 English-French and WMT'17 English-German translation and LibriSpeech speech recognition datasets.

References in corpus (17)

• Efficient Estimation of Word Representations in Vector Space
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• The Curious Case of Neural Text Degeneration
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Non-Autoregressive Neural Machine Translation
• Deep Ensembles: A Loss Landscape Perspective
• Listen, Attend and Spell
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Common Voice: A Massively-Multilingual Speech Corpus
• Learning to Generate Long-term Future via Hierarchical Prediction
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Transformers with convolutional context for ASR
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness
• Analyzing Uncertainty in Neural Machine Translation
• Calibration of Encoder Decoder Models for Neural Machine Translation
• Improving Back-Translation with Uncertainty-based Confidence Estimation

Cited by in corpus (4)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
• What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
• StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation