XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Abstract

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

arXiv commentIn Proceedings of the 37th International Conference on Machine Learning (ICML). July 2020

References in corpus (7)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Exploiting Similarities among Languages for Machine Translation
• On the Cross-lingual Transferability of Monolingual Representations
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• XNLI: Evaluating Cross-lingual Sentence Representations
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation

Cited by in corpus (78)

• On the Opportunities and Risks of Foundation Models
• mT5: A massively multilingual pre-trained text-to-text transformer
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
• MuRIL: Multilingual Representations for Indian Languages
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Pre-training via Paraphrasing
• Translation Artifacts in Cross-lingual Transfer Learning
• KLUE: Korean Language Understanding Evaluation
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition
• Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages
• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
• Multi-Task Learning in Natural Language Processing: An Overview
• Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
• Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond
• Boosting Neural Networks to Decompile Optimized Binaries
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• NewsEmbed: Modeling News through Pre-trained Document Representations
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• NaturalProofs: Mathematical Theorem Proving in Natural Language
• Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
• MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities
• On Learning Universal Representations Across Languages
• Rethinking embedding coupling in pre-trained language models
• Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer
• LAReQA: Language-agnostic answer retrieval from a multilingual pool
• GLGE: A New General Language Generation Evaluation Benchmark
• The Benchmark Lottery
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• Phrase-level Active Learning for Neural Machine Translation
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• Robust Optimization for Multilingual Translation with Imbalanced Data
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• A Multilingual Virtual Guide for Self-Attachment Technique
• Which *BERT? A Survey Organizing Contextualized Encoders
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• Language Anisotropic Cross-Lingual Model Editing
• Multilingual BERT Post-Pretraining Alignment
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• Revisiting Language Encoding in Learning Multilingual Representations
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
• Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
• Problems and Countermeasures in Natural Language Processing Evaluation
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi
• Multi-view Subword Regularization
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
• Cross-language Information Retrieval
• On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning
• Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines
• GEM: A General Evaluation Benchmark for Multimodal Tasks
• Cross-lingual Extended Named Entity Classification of Wikipedia Articles
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation
• Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
• Balancing Average and Worst-case Accuracy in Multitask Learning
• Predicting the Performance of Multilingual NLP Models
• Multilingual AMR-to-Text Generation
• Graph Convolutional Network for Swahili News Classification