Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
arXiv:2003.14056
Abstract
It is now established that modern neural language models can be successfully trained on multiple languages simultaneously without changes to the underlying architecture. But what kind of knowledge is really shared among languages within these models? Does multilingual training mostly lead to an alignment of the lexical representation spaces or does it also enable the sharing of purely grammatical knowledge? In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors, using a variety of models and probing tasks. We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer.
arXiv commentv3: Submitted to NaDaLiDa 2021. 9 pages, two columns and with 6 figures and 1 table
References in corpus (13)
- Cross-lingual Language Model Pretraining
- On the Cross-lingual Transferability of Monolingual Representations
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- What do Neural Machine Translation Models Learn about Morphology?
- XNLI: Evaluating Cross-lingual Sentence Representations
- Theoretical Limitations of Self-Attention in Neural Sequence Models
- How multilingual is Multilingual BERT?
- How Language-Neutral is Multilingual BERT?
- Single Headed Attention RNN: Stop Thinking With Your Head
- Analyzing the Limitations of Cross-lingual Word Embedding Mappings
- Emerging Cross-lingual Structure in Pretrained Language Models
- The Importance of Being Recurrent for Modeling Hierarchical Structure
- Can LSTM Learn to Capture Agreement? The Case of Basque