Cross-lingual Retrieval for Iterative Self-Supervised Training

Abstract

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Exploiting Similarities among Languages for Machine Translation
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• REALM: Retrieval-Augmented Language Model Pre-Training
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Generalization through Memorization: Nearest Neighbor Language Models
• Bilingual Lexicon Induction through Unsupervised Machine Translation
• Emerging Cross-lingual Structure in Pretrained Language Models
• Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models
• Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

Cited by in corpus (6)

• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
• ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models
• Current Limitations of Language Models: What You Need is Retrieval
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data