Encoders Help You Disambiguate Word Senses in Neural Machine Translation
arXiv:1908.11771 · doi:10.18653/v1/d19-1149
Abstract
Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. Decoders could provide further relevant information for disambiguation. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. Note that this is a revised version. The content related to decoder hidden states has been updated.
arXiv commentUpdate with corrections. Here is the link to the erratum: https://www.aclweb.org/anthology/D19-1149e1.pdf
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- What do Neural Machine Translation Models Learn about Morphology?
- Sockeye: A Toolkit for Neural Machine Translation
- Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
- Encoders Help You Disambiguate Word Senses in Neural Machine Translation
Cited by in corpus (6)
- Encoders Help You Disambiguate Word Senses in Neural Machine Translation
- On the Sub-Layer Functionalities of Transformer Decoder
- Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English
- Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers
- Rethinking the Value of Transformer Components
- On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation