Explicit Alignment Objectives for Multilingual Bidirectional Encoders

Abstract

Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) have proven to be impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model which has 3.2x the parameters of AMBER. Our code and models are available at http://github.com/junjiehu/amber.

arXiv commentpublished at NAACL 2021

References in corpus (6)

• On the Cross-lingual Transferability of Monolingual Representations
• Dual Learning for Machine Translation
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation

Cited by in corpus (2)

• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment