A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
arXiv:2004.09205
Abstract
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability. We also observe the language-specific information in multilingual BERT. By manipulating the latent representations, we can control the output languages of multilingual BERT, and achieve unsupervised token translation. We further show that based on the observation, there is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
References in corpus (8)
- Distributed Representations of Words and Phrases and their Compositionality
- Exploiting Similarities among Languages for Machine Translation
- Massively Multilingual Word Embeddings
- Offline bilingual word vectors, orthogonal transformations and the inverted softmax
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Multilingual Alignment of Contextual Word Representations
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- From English To Foreign Languages: Transferring Pre-trained Language Models