CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
arXiv:1811.00937
Abstract
When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.
arXiv commentaccepted as a long paper at NAACL 2019
References in corpus (10)
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- Enhanced LSTM for Natural Language Inference
- Exploring the Limits of Language Modeling
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Deep Learning for Answer Sentence Selection
- A Simple Method for Commonsense Reasoning
- Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
- MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Cited by in corpus (33)
- Unleashing the potential of prompt engineering for large language models
- An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Exploring the State of the Art in Legal QA Systems
- Explaining Question Answering Models through Text Generation
- Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
- Unifying Question Answering, Text Classification, and Regression via Span Extraction
- Training Verifiers to Solve Math Word Problems
- On Grounded Planning for Embodied Tasks with Language Models
- Natural Language Processing with Commonsense Knowledge: A Survey
- Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach
- Pre-training Text-to-Text Transformers for Concept-centric Common Sense
- GNN is a Counter? Revisiting GNN for Question Answering
- Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering
- Consolidating Commonsense Knowledge
- Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation
- Fusing Context Into Knowledge Graph for Commonsense Question Answering
- Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
- An Enhanced Knowledge Injection Model for Commonsense Generation
- UnCommonSense: Informative Negative Knowledge about Everyday Concepts
- AR-LSAT: Investigating Analytical Reasoning of Text
- Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation
- Generalizable Neuro-symbolic Systems for Commonsense Question Answering
- ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks
- Natural Language QA Approaches using Reasoning with External Knowledge
- Target-driven Attack for Large Language Models
- A World-Self Model Towards Understanding Intelligence
- Commonsense knowledge adversarial dataset that challenges ELECTRA
- Analyzing Multi-Task Learning for Abstractive Text Summarization
- Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
- Understanding in Artificial Intelligence
- Beyond Language: Learning Commonsense from Images for Reasoning
- Experience and Prediction: A Metric of Hardness for a Novel Litmus Test