MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
arXiv:2104.08692
Abstract
Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on eight multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.
EMNLP 2021
References in corpus (7)
- On the Cross-lingual Transferability of Monolingual Representations
- Multilingual Denoising Pre-training for Neural Machine Translation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Multilingual Alignment of Contextual Word Representations
- Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks