From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
arXiv:1909.01958
Abstract
AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy, but the rich variety of standardized exams has remained a landmark challenge. Even in 2016, the best AI system achieved merely 59.3% on an 8th Grade science exam challenge. This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern NLP methods can result in mastery on this task. While not a full solution to general question-answering (the questions are multiple choice, and the domain is restricted to 8th Grade science), it represents a significant milestone for the field.
arXiv commentAI Magazine 41 (4) Winter 2020. New analysis sections added
References in corpus (6)
- Bidirectional Attention Flow for Machine Comprehension
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Question Answering via Integer Programming over Semi-Structured Knowledge
- Question Answering as Global Reasoning over Semantic Abstractions
- Reasoning about Actions and State Changes by Injecting Commonsense Knowledge
- Leveraging Term Banks for Answering Complex Questions: A Case for Sparse Vectors
Cited by in corpus (11)
- The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
- Knowledge-Aware Language Model Pretraining
- GNN is a Counter? Revisiting GNN for Question Answering
- Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering
- A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
- Fusing Context Into Knowledge Graph for Commonsense Question Answering
- ParsiNLU: A Suite of Language Understanding Challenges for Persian
- On Controllability of AI
- Challenge Closed-book Science Exam: A Meta-learning Based Question Answering System
- Autoregressive Reasoning over Chains of Facts with Transformers