Multilingual BERT Post-Pretraining Alignment
arXiv:2010.12547
Abstract
We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves comparable result to XLM for translate-train while using less than 18% of the same parallel data and 31% less model parameters. On MLQA, our model outperforms XLM-R_Base that has 57% more parameters than ours.
arXiv commentAccepted at NAACL2021
References in corpus (9)
- A Simple Framework for Contrastive Learning of Visual Representations
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Multilingual Alignment of Contextual Word Representations
- XNLI: Evaluating Cross-lingual Sentence Representations
- How multilingual is Multilingual BERT?
- Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
- On Learning Universal Representations Across Languages