Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
arXiv:2011.03863
Abstract
Recent developments in pre-trained neural language modeling have led to leaps in accuracy on commonsense question-answering benchmarks. However, there is increasing concern that models overfit to specific tasks, without learning to utilize external knowledge or perform general semantic reasoning. In contrast, zero-shot evaluations have shown promise as a more robust measure of a model's general reasoning abilities. In this paper, we propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre-training models. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. Extending on prior work, we devise and compare four constrained distractor-sampling strategies. We provide empirical results across five commonsense question-answering tasks with data generated from five external knowledge resources. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks. In addition, both preserving the structure of the task as well as generating fair and informative questions help language models learn more effectively.
arXiv commentAAAI 2021
References in corpus (12)
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Language Models as Knowledge Bases?
- A Surprisingly Robust Trick for Winograd Schema Challenge
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- HellaSwag: Can a Machine Really Finish Your Sentence?
- Abductive Commonsense Reasoning
- How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
- Evaluating Commonsense in Pre-trained Language Models
- Consolidating Commonsense Knowledge
- A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction