Confidence through Attention

Abstract

Attention distributions of the generated translations are a useful bi-product of attention-based recurrent neural network translation models and can be treated as soft alignments between the input and output tokens. In this work, we use attention distributions as a confidence metric for output translations. We present two strategies of using the attention distributions: filtering out bad translations from a large back-translated corpus, and selecting the best translation in a hybrid setup of two different translation systems. While manual evaluation indicated only a weak correlation between our confidence score and human judgments, the use-cases showed improvements of up to 2.22 BLEU points for filtering and 0.99 points for hybrid translation, tested on English<->German and English<->Latvian translation.

References in corpus (5)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Modeling Coverage for Neural Machine Translation

Cited by in corpus (4)

• Improving Back-Translation with Uncertainty-based Confidence Estimation
• Sparse Attention with Linear Units
• Neural Machine Translation: A Review and Survey
• Revisiting Negation in Neural Machine Translation