Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

Abstract

Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.

arXiv comment11 pages, 5 figures, accepted by EMNLP 2018 (v2: corrected author names; v3: fix to CNN context-window size, and new post-publication experiments in section 6)

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Comparative Study of CNN and RNN for Natural Language Processing
• Sockeye: A Toolkit for Neural Machine Translation
• Scale-Invariant Convolutional Neural Networks
• The Importance of Being Recurrent for Modeling Hierarchical Structure

Cited by in corpus (13)

• Context-Aware Self-Attention Networks
• Context in Neural Machine Translation: A Review of Models and Evaluations
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• What Do You Mean I'm Funny? Personalizing the Joke Skill of a Voice-Controlled Virtual Assistant
• Improving Neural Machine Translation with Pre-trained Representation
• Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
• DTMT: A Novel Deep Transition Architecture for Neural Machine Translation
• Fine Grained Human Evaluation for English-to-Chinese Machine Translation: A Case Study on Scientific Text
• Meta-Embeddings Based On Self-Attention
• An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures
• NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
• Dependency-Based Neural Representations for Classifying Lines of Programs
• Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning