Calibration of Encoder Decoder Models for Neural Machine Translation
arXiv:1903.00802
Abstract
We study the calibration of several state of the art neural machine translation(NMT) systems built on attention-based encoder-decoder models. For structured outputs like in NMT, calibration is important not just for reliable confidence with predictions, but also for proper functioning of beam-search inference. We show that most modern NMT models are surprisingly miscalibrated even when conditioned on the true previous tokens. Our investigation leads to two main reasons -- severe miscalibration of EOS (end of sequence marker) and suppression of attention uncertainty. We design recalibration methods based on these signals and demonstrate improved accuracy, better sequence-level calibration, and more intuitive results from beam-search.
12 Pages
References in corpus (6)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- On Fairness and Calibration
- Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
Cited by in corpus (13)
- When Does Label Smoothing Help?
- Uncertainty Estimation in Autoregressive Structured Prediction
- Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
- Decoding and Diversity in Machine Translation
- On the Inference Calibration of Neural Machine Translation
- Temporal Probability Calibration
- Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation
- Folden: -Fold Ensemble for Out-Of-Distribution Detection
- Neural Machine Translation: A Review and Survey
- Smoothing and Shrinking the Sparse Seq2Seq Search Space
- Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation
- Knowing More About Questions Can Help: Improving Calibration in Question Answering
- Calibrating Structured Output Predictors for Natural Language Processing