Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

Yuan Chai, Yaobo Liang, Nan Duan

Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (M…