Quasar: Datasets for Question Answering by Search and Reading

Abstract

We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at https://github.com/bdhingra/quasar .

Cited by in corpus (53)

• Passage Re-ranking with BERT
• Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Rethinking Search: Making Domain Experts out of Dilettantes
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
• Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks
• Open-Retrieval Conversational Question Answering
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Interactive Question Answering Systems: Literature Review
• Universal Text Representation from BERT: An Empirical Study
• Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models
• PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
• Improving Question Answering with External Knowledge
• Combining Fact Extraction and Verification with Neural Semantic Matching Networks
• Neural Machine Reading Comprehension: Methods and Trends
• Efficient and Robust Question Answering from Minimal Context over Documents
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
• A Survey of Knowledge Enhanced Pre-trained Models
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• Unsupervised Domain Adaptation of Language Models for Reading Comprehension
• Multi-Paragraph Reasoning with Knowledge-enhanced Graph Neural Network
• Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering
• Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text
• Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• Knowledge-Aided Open-Domain Question Answering
• More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• Controlling Risk of Web Question Answering
• Quizbowl: The Case for Incremental Question Answering
• HAS-QA: Hierarchical Answer Spans Model for Open-domain Question Answering
• Training a Ranking Function for Open-Domain Question Answering
• Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
• A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction
• Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension
• A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases
• Tackling Graphical NLP problems with Graph Recurrent Networks
• Keyword-based Query Comprehending via Multiple Optimized-Demand Augmentation
• MRNN: A Multi-Resolution Neural Network with Duplex Attention for Document Retrieval in the Context of Question Answering
• Weakly-Supervised Open-Retrieval Conversational Question Answering
• Knowledge-Empowered Representation Learning for Chinese Medical Reading Comprehension: Task, Model and Resources
• Cross-Thought for Sentence Encoder Pre-training
• Syntax-Enhanced Pre-trained Model
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
• Knowledge Efficient Deep Learning for Natural Language Processing
• Cross-language Sentence Selection via Data Augmentation and Rationale Training
• WebSRC: A Dataset for Web-Based Structural Reading Comprehension
• A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
• How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering