ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training

Henry Tang, Ameet Deshpande, Karthik Narasimhan

Multilingual pre-trained models exhibit zero-shot cross-lingual transfer, where a model fine-tuned on a source language achieves surprisingly good performance on a target language.…