LAReQA: Language-agnostic answer retrieval from a multilingual pool

Abstract

We present LAReQA, a challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool. Unlike previous cross-lingual tasks, LAReQA tests for "strong" cross-lingual alignment, requiring semantically related cross-language pairs to be closer in representation space than unrelated same-language pairs. Building on multilingual BERT (mBERT), we study different strategies for achieving strong alignment. We find that augmenting training data via machine translation is effective, and improves significantly over using mBERT out-of-the-box. Interestingly, the embedding baseline that performs the best on LAReQA falls short of competing baselines on zero-shot variants of our task that only target "weak" alignment. This finding underscores our claim that languageagnostic retrieval is a substantively new kind of cross-lingual evaluation.

References in corpus (7)

• On the Cross-lingual Transferability of Monolingual Representations
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Efficient Natural Language Response Suggestion for Smart Reply
• XNLI: Evaluating Cross-lingual Sentence Representations
• End-to-End Retrieval in Continuous Space
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

Cited by in corpus (4)

• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability
• MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
• Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation