Emerging Cross-lingual Structure in Pretrained Language Models

Abstract

We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models are so effective for cross-lingual transfer. We show, contrary to what was previously hypothesized, that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains. The only requirement is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from independently trained models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces. For multilingual masked language modeling, these symmetries seem to be automatically discovered and aligned during the joint training process.

arXiv commentACL 2020

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Gaussian Error Linear Units (GELUs)
• Cross-lingual Language Model Pretraining
• Exploiting Similarities among Languages for Machine Translation
• On the Cross-lingual Transferability of Monolingual Representations
• Similarity of Neural Network Representations Revisited
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Word Translation Without Parallel Data
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Convergent Learning: Do different neural networks learn the same representations?
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Analyzing the Limitations of Cross-lingual Word Embedding Mappings
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation

Cited by in corpus (10)

• On the Cross-lingual Transferability of Monolingual Representations
• Do Vision Transformers See Like Convolutional Neural Networks?
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Are All Languages Created Equal in Multilingual BERT?
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• A Survey of Syntactic-Semantic Parsing Based on Constituent and Dependency Structures
• Mining Knowledge for Natural Language Inference from Wikipedia Categories