XNLI: Evaluating Cross-lingual Sentence Representations

Abstract

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

arXiv commentEMNLP 2018

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Exploiting Similarities among Languages for Machine Translation
• Universal Sentence Encoder
• A large annotated corpus for learning natural language inference
• Massively Multilingual Word Embeddings
• Natural Language Inference over Interaction Space
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification
• Multilingual Models for Compositional Distributed Semantics
• Learning Bilingual Word Representations by Marginalizing Alignments

Cited by in corpus (84)

• Cross-lingual Language Model Pretraining
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Multilingual Denoising Pre-training for Neural Machine Translation
• mT5: A massively multilingual pre-trained text-to-text transformer
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Entailment as Few-Shot Learner
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• KLUE: Korean Language Understanding Evaluation
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• FlauBERT: Unsupervised Language Model Pre-training for French
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• A Survey of Code-switched Speech and Language Processing
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
• Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• Unifying Question Answering, Text Classification, and Regression via Span Extraction
• From English To Foreign Languages: Transferring Pre-trained Language Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• A New Dataset for Natural Language Inference from Code-mixed Conversations
• Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference
• KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding
• Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog
• Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus
• On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning
• Pre-Trained Models: Past, Present and Future
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Bridging the domain gap in cross-lingual document classification
• ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
• Multilingual Question Answering from Formatted Text applied to Conversational Agents
• Studying Politeness across Cultures Using English Twitter and Mandarin Weibo
• On Learning Universal Representations Across Languages
• Cross-Lingual Natural Language Generation via Pre-Training
• Is Machine Learning Speaking my Language? A Critical Look at the NLP-Pipeline Across 8 Human Languages
• DaNetQA: a yes/no Question Answering Dataset for the Russian Language
• LAReQA: Language-agnostic answer retrieval from a multilingual pool
• Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Unsupervised Domain Adaptation with Adapter
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• Evaluation of Sentence Representations in Polish
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• Multilingual BERT Post-Pretraining Alignment
• Meta-Learning with MAML on Trees
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• Revisiting Language Encoding in Learning Multilingual Representations
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Towards Interlingua Neural Machine Translation
• Cross-lingual Inductive Transfer to Detect Offensive Language
• Multi-view Subword Regularization
• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
• Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
• Aligning Cross-Lingual Entities with Multi-Aspect Information
• LSCP: Enhanced Large Scale Colloquial Persian Language Understanding
• Extracting and filtering paraphrases by bridging natural language inference and paraphrasing
• RuSentEval: Linguistic Source, Encoder Force!
• Unsupervised Word Translation Pairing using Refinement based Point Set Registration
• Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings
• Predicting the Performance of Multilingual NLP Models
• Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines
• On the Universality of Deep Contextual Language Models
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• Cross-Lingual Low-Resource Set-to-Description Retrieval for Global E-Commerce
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals