TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

Abstract

Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA---a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology---the set of linguistic features each language expresses---such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don't know the answer yet, and the data is collected directly in each language without the use of translation.

arXiv commentTo appear in Transactions of the Association for Computational Linguistics (TACL) 2020. Please use this as the citation

References in corpus (14)

• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• On the Cross-lingual Transferability of Monolingual Representations
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• XNLI: Evaluating Cross-lingual Sentence Representations
• A BERT Baseline for the Natural Questions
• Reading Wikipedia to Answer Open-Domain Questions
• DRCD: a Chinese Machine Reading Comprehension Dataset
• Multilingual Extractive Reading Comprehension by Runtime Machine Translation
• QuAC : Question Answering in Context
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
• KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension
• Question Answering is a Format; When is it Useful?
• A Generative Approach to Question Answering

Cited by in corpus (8)

• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks
• On Learning Universal Representations Across Languages
• Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages
• LAReQA: Language-agnostic answer retrieval from a multilingual pool
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• IIRC: A Dataset of Incomplete Information Reading Comprehension Questions