What makes multilingual BERT multilingual?
arXiv:2010.10938
Abstract
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.
arXiv admin note: substantial text overlap with arXiv:2004.09205
References in corpus (5)
- Offline bilingual word vectors, orthogonal transformations and the inverted softmax
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Multilingual Alignment of Contextual Word Representations
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- From English To Foreign Languages: Transferring Pre-trained Language Models