Analyzing Uncertainty in Neural Machine Translation

Abstract

Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. As part of this study, we release multiple human reference translations for two popular benchmarks.

arXiv commentICML 2018

References in corpus (8)

• Convolutional Sequence to Sequence Learning
• On Calibration of Modern Neural Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Mutual Information and Diverse Decoding Improve Neural Machine Translation
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Variational Neural Machine Translation
• Analyzing Neural MT Search and Model Performance

Cited by in corpus (20)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• When Does Label Smoothing Help?
• Language GANs Falling Short
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Uncertainty Estimation in Autoregressive Structured Prediction
• Trading Off Diversity and Quality in Natural Language Generation
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation
• Discriminative Adversarial Search for Abstractive Summarization
• Mixture Content Selection for Diverse Sequence Generation
• On the Inference Calibration of Neural Machine Translation
• Capturing Label Distribution: A Case Study in NLI
• Extractive Summary as Discrete Latent Variables
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation
• Data Troubles in Sentence Level Confidence Estimation for Machine Translation
• Accelerated Reinforcement Learning for Sentence Generation by Vocabulary Prediction
• Generating Diverse Translation by Manipulating Multi-Head Attention
• Generating Diverse Translation from Model Distribution with Dropout
• Quantum Statistics-Inspired Neural Attention