Cross-Lingual Language Model Meta-Pretraining

Abstract

The success of pretrained cross-lingual language models relies on two essential abilities, i.e., generalization ability for learning downstream tasks in a source language, and cross-lingual transferability for transferring the task knowledge to other languages. However, current methods jointly learn the two abilities in a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer. In this paper, we propose cross-lingual language model meta-pretraining, which learns the two abilities in different training phases. Our method introduces an additional meta-pretraining phase before cross-lingual pretraining, where the model learns generalization ability on a large-scale monolingual corpus. Then, the model focuses on learning cross-lingual transfer on a multilingual corpus. Experimental results show that our method improves both generalization and cross-lingual transfer, and produces better-aligned representations across different languages.

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Multilingual Denoising Pre-training for Neural Machine Translation
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• On the ability of monolingual models to learn language-agnostic representations