Quizbowl: The Case for Incremental Question Answering
arXiv:1904.04792
Abstract
Scholastic trivia competitions test knowledge and intelligence through mastery of question answering. Modern question answering benchmarks are one variant of the Turing test. Specifically, answering a set of questions as well as a human is a minimum bar towards demonstrating human-like intelligence. This paper makes the case that the format of one competition -- where participants can answer in the middle of hearing a question (incremental) -- better differentiates the skill between (human or machine) players. Additionally, merging a sequential decision-making sub-task with question answering (QA) provides a good setting for research in model calibration and opponent modeling. Thus, embedded in this task are three machine learning challenges: (1) factoid QA over thousands of Wikipedia-like answers, (2) calibration of the QA model's confidence scores, and (3) sequential decision-making that incorporates knowledge of the QA model, its calibration, and what the opponent may do. We make two contributions: (1) collecting and curating a large factoid QA dataset and an accompanying gameplay dataset, and (2) developing a model that addresses these three machine learning challenges. In addition to offline evaluation, we pitted our model against some of the most accomplished trivia players in the world in a series of exhibition matches spanning several years. Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Gaussian Error Linear Units (GELUs)
- Teaching Machines to Read and Comprehend
- On Calibration of Modern Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Frustratingly Easy Domain Adaptation
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Large-scale Simple Question Answering with Memory Networks
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- To Trust Or Not To Trust A Classifier
- A Reliable Effective Terascale Linear Learning System
- Pathologies of Neural Models Make Interpretations Difficult
- Natural Adversarial Examples
- Quasar: Datasets for Question Answering by Search and Reading
- Logarithmic Time One-Against-Some
- Question Answering is a Format; When is it Useful?
- Compositional Questions Do Not Necessitate Multi-hop Reasoning
- Estimating Uncertainty Online Against an Adversary