NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
arXiv:2101.00133
Abstract
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.
arXiv comment26 pages; Published in Proceedings of Machine Learning Research (PMLR), NeurIPS 2020 Competition and Demonstration Track
References in corpus (12)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Longformer: The Long-Document Transformer
- REALM: Retrieval-Augmented Language Model Pre-Training
- Unsupervised Question Answering by Cloze Translation
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
- Distilling Knowledge from Reader to Retriever for Question Answering
- Generation-Augmented Retrieval for Open-domain Question Answering
- A Memory Efficient Baseline for Open Domain Question Answering
- Is Retriever Merely an Approximator of Reader?
- ELI5: Long Form Question Answering
- Quizbowl: The Case for Incremental Question Answering
Cited by in corpus (14)
- Generation-Augmented Retrieval for Open-domain Question Answering
- Cross-Domain Aspect Extraction using Transformers Augmented with Knowledge Graphs
- Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
- Utility is in the Eye of the User: A Critique of NLP Leaderboards
- Efficient Nearest Neighbor Language Models
- Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
- UnitedQA: A Hybrid Approach for Open Domain Question Answering
- BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
- Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
- Evaluation of Semantic Answer Similarity Metrics
- A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation
- Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
- Highly Parallel Autoregressive Entity Linking with Discriminative Correction
- What's in a Name? Answer Equivalence For Open-Domain Question Answering