A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages

Abstract

This work describes experiments which probe the hidden representations of several BERT-style models for morphological content. The goal is to examine the extent to which discrete linguistic structure, in the form of morphological features and feature values, presents itself in the vector representations and attention distributions of pre-trained language models for five European languages. The experiments contained herein show that (i) Transformer architectures largely partition their embedding space into convex sub-regions highly correlated with morphological feature value, (ii) the contextualized nature of transformer embeddings allows models to distinguish ambiguous morphological forms in many, but not all cases, and (iii) very specific attention head/layer combinations appear to hone in on subject-verb agreement.

References in corpus (9)

• CamemBERT: a Tasty French Language Model
• Assessing BERT's Syntactic Abilities
• What do Neural Machine Translation Models Learn about Morphology?
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• Visualizing and Measuring the Geometry of BERT
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• Can LSTM Learn to Capture Agreement? The Case of Basque
• Dissecting Contextual Word Embeddings: Architecture and Representation

Cited by in corpus (5)

• Enhancing deep neural networks with morphological information
• Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
• Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation
• Are pre-trained text representations useful for multilingual and multi-dimensional language proficiency modeling?
• Rule-based Morphological Inflection Improves Neural Terminology Translation