Adding Interpretable Attention to Neural Translation Models Improves Word Alignment

Abstract

Multi-layer models with multiple attention heads per layer provide superior translation quality compared to simpler and shallower models, but determining what source context is most relevant to each target word is more challenging as a result. Therefore, deriving high-accuracy word alignments from the activations of a state-of-the-art neural machine translation model is an open challenge. We propose a simple model extension to the Transformer architecture that makes use of its hidden representations and is restricted to attend solely on encoder information to predict the next word. It can be trained on bilingual data without word-alignment information. We further introduce a novel alignment inference procedure which applies stochastic gradient descent to directly optimize the attention activations towards a given target word. The resulting alignments dramatically outperform the naive approach to interpreting Transformer attention activations, and are comparable to Giza++ on two publicly available data sets.

References in corpus (1)

• Sequence to Sequence Learning with Neural Networks

Cited by in corpus (19)

• On Identifiability in Transformers
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Word Alignment by Fine-tuning Embeddings on Parallel Corpora
• Novel Applications of Factored Neural Machine Translation
• Saliency-driven Word Alignment Interpretation for Neural Machine Translation
• Attention as a Guide for Simultaneous Speech Translation
• A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
• Evaluating Explanation Methods for Neural Machine Translation
• Sparse Attention with Linear Units
• Embedding-Enhanced Giza++: Improving Alignment in Low- and High- Resource Scenarios Using Embedding Space Geometry
• Mask-Align: Self-Supervised Neural Word Alignment
• Accurate Word Alignment Induction from Neural Machine Translation
• Neural Machine Translation: A Review and Survey
• Machine Translation with Unsupervised Length-Constraints
• Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment
• Jointly Learning to Align and Translate with Transformer Models
• A Discriminative Neural Model for Cross-Lingual Word Alignment
• Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions
• Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models