What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties

Abstract

Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that they encode. We propose methods for probing sentence representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological and syntactic structure. In addition, we investigate how this information is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• What do Neural Machine Translation Models Learn about Morphology?
• Massively Multilingual Word Embeddings
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• XNLI: Evaluating Cross-lingual Sentence Representations
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• What the [MASK]? Making Sense of Language-Specific BERT Models
• How Language-Neutral is Multilingual BERT?
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers

Cited by in corpus (7)

• Locating Language-Specific Information in Contextualized Embeddings
• Are Pretrained Multilingual Models Equally Fair Across Languages?
• Cross-neutralising: Probing for joint encoding of linguistic information in multilingual models
• Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models
• Discovering Representation Sprachbund For Multilingual Pre-Training
• RuSentEval: Linguistic Source, Encoder Force!
• On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning