Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
arXiv:2010.10041
Abstract
Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information. We find that the representation of a language can be obtained by simply averaging the embeddings of the tokens of the language. Given this language representation, we control the output languages of multilingual BERT by manipulating the token embeddings, thus achieving unsupervised token translation. We further propose a computationally cheap but effective approach to improve the cross-lingual ability of m-BERT based on this observation.
preprint
References in corpus (5)
- Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Multilingual Alignment of Contextual Word Representations
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- What makes multilingual BERT multilingual?