Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
arXiv:1801.07772
Abstract
While neural machine translation (NMT) models provide improved translation quality in an elegant, end-to-end framework, it is less clear what they learn about language. Recent work has started evaluating the quality of vector representations learned by NMT models on morphological and syntactic tasks. In this paper, we investigate the representations learned at different layers of NMT encoders. We train NMT systems on parallel data and use the trained models to extract features for training a classifier on two tasks: part-of-speech and semantic tagging. We then measure the performance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding representation learning in NMT models. For instance, we find that higher layers are better at learning semantics while lower layers tend to be better for part-of-speech tagging. We also observe little effect of the target language on source-side representations, especially with higher quality NMT models.
arXiv commentIJCNLP 2017
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Visualizing and Understanding Recurrent Networks
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- Semantic Tagging with Deep Residual Networks
- Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries
- Representation of linguistic form and function in recurrent neural networks
Cited by in corpus (4)
- On the Effect of Dropping Layers of Pre-trained Transformer Models
- Encoders Help You Disambiguate Word Senses in Neural Machine Translation
- Does injecting linguistic structure into language models lead to better alignment with brain recordings?
- Is neural language acquisition similar to natural? A chronological probing study