What do Neural Machine Translation Models Learn about Morphology?

Abstract

Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture. However, little is known about what these models learn about source and target languages during the training process. In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.

arXiv commentUpdated decoder experiments

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Visualizing and Understanding Recurrent Networks
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• A Hybrid Morpheme-Word Representation for Machine Translation of Morphologically Rich Languages
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• Representation of linguistic form and function in recurrent neural networks
• Large-Scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results
• An Arabic-Hebrew parallel corpus of TED talks

Cited by in corpus (68)

• On the Opportunities and Risks of Foundation Models
• Pre-trained Models for Natural Language Processing: A Survey
• Synthetic and Natural Noise Both Break Neural Machine Translation
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis
• Local Interpretations for Explainable Natural Language Processing: A Survey
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
• Measuring and Improving Consistency in Pretrained Language Models
• HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• A Systematic Assessment of Syntactic Generalization in Neural Language Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• Encoders Help You Disambiguate Word Senses in Neural Machine Translation
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• Lost in Translation: Analysis of Information Loss During Machine Translation Between Polysynthetic and Fusional Languages
• A Linguistic Study on Relevance Modeling in Information Retrieval
• How Reliable are Model Diagnostics?
• Morphosyntactic probing of multilingual BERT models
• What does Attention in Neural Machine Translation Pay Attention to?
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• Towards Understanding Neural Machine Translation with Word Importance
• Dynamic Multi-Level Multi-Task Learning for Sentence Simplification
• Probing Neural Dialog Models for Conversational Understanding
• Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction
• What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
• Measuring Memorization Effect in Word-Level Neural Networks Probing
• Do latent tree learning models identify meaningful structure in sentences?
• BPE and CharCNNs for Translation of Morphology: A Cross-Lingual Comparison and Analysis
• Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
• Dissecting Contextual Word Embeddings: Architecture and Representation
• On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation
• Assessing the Ability of Self-Attention Networks to Learn Word Order
• Discovery of Natural Language Concepts in Individual Units of CNNs
• Grounding Representation Similarity with Statistical Testing
• It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Computational Morphology with Neural Network Approaches
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• On convex decision regions in deep network representations
• How Does Adversarial Fine-Tuning Benefit BERT?
• Inducing Grammars with and for Neural Machine Translation
• Survey: Transformer based Video-Language Pre-training
• Effect of Post-processing on Contextualized Word Representations
• Towards a Universal Continuous Knowledge Base
• Assessing the Bilingual Knowledge Learned by Neural Machine Translation Models
• On the Sub-Layer Functionalities of Transformer Decoder
• Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition
• Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• What do character-level models learn about morphology? The case of dependency parsing
• An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures
• Fast Model Editing at Scale
• Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
• Evaluating Document Coherence Modelling
• Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
• Character-Level Models versus Morphology in Semantic Role Labeling
• diagNNose: A Library for Neural Activation Analysis
• Bi-Decoder Augmented Network for Neural Machine Translation
• COSTRA 1.0: A Dataset of Complex Sentence Transformations
• Variation and generality in encoding of syntactic anomaly information in sentence embeddings
• Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models
• Investigating context features hidden in End-to-End TTS
• Fine-grained Interpretation and Causation Analysis in Deep NLP Models
• Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification