mT5: A massively multilingual pre-trained text-to-text transformer

Abstract

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

References in corpus (18)

• Cross-lingual Language Model Pretraining
• CamemBERT: a Tasty French Language Model
• On the Cross-lingual Transferability of Monolingual Representations
• The Natural Language Decathlon: Multitask Learning as Question Answering
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• BERTje: A Dutch BERT Model
• XNLI: Evaluating Cross-lingual Sentence Representations
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Pre-training via Paraphrasing
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Playing with Words at the National Library of Sweden -- Making a Swedish BERT
• Unifying Question Answering, Text Classification, and Regression via Span Extraction
• GLU Variants Improve Transformer
• Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Rethinking embedding coupling in pre-trained language models
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data

Cited by in corpus (57)

• BEiT: BERT Pre-Training of Image Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Rethinking Search: Making Domain Experts out of Dilettantes
• IndicBART: A Pre-trained Model for Indic Natural Language Generation
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
• Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer
• Overview of BioASQ 2020: The eighth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering
• Improving Word Translation via Two-Stage Contrastive Learning
• RuCoLA: Russian Corpus of Linguistic Acceptability
• BLP-2023 Task 2: Sentiment Analysis
• Specializing Multilingual Language Models: An Empirical Study
• Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages
• KinyaBERT: a Morphology-aware Kinyarwanda Language Model
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
• XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
• Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
• Code-switched inspired losses for generic spoken dialog representations
• Abstractive Text Summarization Using the BRIO Training Paradigm
• Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
• Modelling Latent Translations for Cross-Lingual Transfer
• DaCy: A Unified Framework for Danish NLP
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• TADA: Task-Agnostic Dialect Adapters for English
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Multimedia Generative Script Learning for Task Planning
• Language Model Augmented Relevance Score
• Language Anisotropic Cross-Lingual Model Editing
• Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?
• GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
• MT3: Multi-Task Multitrack Music Transcription
• ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution
• Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
• DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue
• You Can Do Better! If You Elaborate the Reason When Making Prediction
• The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents
• MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Predicting the Performance of Multilingual NLP Models
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• A Dataset and Baselines for Multilingual Reply Suggestion
• Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation
• Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoders
• DEPTH: Discourse Education through Pre-Training Hierarchically
• Simple and Effective Input Reformulations for Translation
• Calculating Question Similarity is Enough: A New Method for KBQA Tasks
• On the Universality of Deep Contextual Language Models
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
• Exploring a Unified Sequence-To-Sequence Transformer for Medical Product Safety Monitoring in Social Media
• Generating abstractive summaries of Lithuanian news articles using a transformer model