Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
arXiv:1803.05457
Abstract
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
10 pages, 7 tables, 2 figures
References in corpus (5)
Cited by in corpus (106)
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Unleashing the potential of prompt engineering for large language models
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
- QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
- Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
- ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
- Generative Data Augmentation for Commonsense Reasoning
- How Context Affects Language Models' Factual Predictions
- Finetuned Language Models Are Zero-Shot Learners
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- Interactive Question Answering Systems: Literature Review
- Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
- Explaining Question Answering Models through Text Generation
- GenericsKB: A Knowledge Base of Generic Statements
- How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
- Developing ChemDFM as a large language foundation model for chemistry
- Advanced Semantics for Commonsense Knowledge Extraction
- General-Purpose Question-Answering with Macaw
- A Survey on Explainability in Machine Reading Comprehension
- How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
- Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension
- Improving Question Answering with External Knowledge
- Domain-specific ChatBots for Science using Embeddings
- Machine Reading Comprehension: a Literature Review
- Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
- Taming Sparsely Activated Transformer with Stochastic Experts
- Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering
- From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
- KG^2: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings
- A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
- Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
- Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
- Red Dragon AI at TextGraphs 2019 Shared Task: Language Model Assisted Explanation Generation
- Unsupervised Domain Adaptation of Language Models for Reading Comprehension
- Improving Question Answering by Commonsense-Based Pre-Training
- Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
- Tucano: Advancing Neural Text Generation for Portuguese
- Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering
- Interactive Language Learning by Question Answering
- Unified Question Generation with Continual Lifelong Learning
- Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
- A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
- A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
- Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
- Read, Retrospect, Select: An MRC Framework to Short Text Entity Linking
- Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
- ParsiNLU: A Suite of Language Understanding Challenges for Persian
- Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning
- ExplanationLP: Abductive Reasoning for Explainable Science Question Answering
- Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
- Cross-Lingual Transfer Learning for Question Answering
- Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering
- NLPGym -- A toolkit for evaluating RL agents on Natural Language Processing Tasks
- HEAD-QA: A Healthcare Dataset for Complex Reasoning
- Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
- Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering
- Mosaic: Composite Projection Pruning for Resource-efficient LLMs
- Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
- Natural Language QA Approaches using Reasoning with External Knowledge
- Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
- cosmosage: A Natural-Language Assistant for Cosmologists
- Dynamic Semantic Graph Construction and Reasoning for Explainable Multi-hop Science Question Answering
- Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks
- Reading Comprehension as Natural Language Inference: A Semantic Analysis
- Case-Based Abductive Natural Language Inference
- Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
- AmazonQA: A Review-Based Question Answering Task
- Encoding Explanatory Knowledge for Zero-shot Science Question Answering
- Repurposing Entailment for Multi-Hop Question Answering Tasks
- Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
- Querent Intent in Multi-Sentence Questions
- JEC-QA: A Legal-Domain Question Answering Dataset
- Large-scale Cloze Test Dataset Created by Teachers
- Complementary Evidence Identification in Open-Domain Question Answering
- Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
- Answering questions by learning to rank -- Learning to rank by answering questions
- SemEval-2020 Task 4: Commonsense Validation and Explanation
- CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
- Extending Automated Deduction for Commonsense Reasoning
- MuTual: A Dataset for Multi-Turn Dialogue Reasoning
- Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
- RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge
- Designing Multimodal Datasets for NLP Challenges
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
- Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores
- Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
- QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
- What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
- LiveQA: A Question Answering Dataset over Sports Live
- ScienceExamCER: A High-Density Fine-Grained Science-Domain Corpus for Common Entity Recognition
- Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
- Reasoning-Driven Question-Answering for Natural Language Understanding
- Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation
- SCIMAT: Science and Mathematics Dataset
- Searching Personal Collections
- Dynamic Framework for Collaborative Learning: Leveraging Advanced LLM with Adaptive Feedback Mechanisms
- Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts
- Towards an Atlas of Cultural Commonsense for Machine Reasoning
- Learning to Generalize for Sequential Decision Making
- Building a Question and Answer System for News Domain
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- What do we expect from Multiple-choice QA Systems?
- Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data