HellaSwag: Can a Machine Really Finish Your Sentence?
arXiv:1905.07830
Abstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
ACL 2019. Project page at https://rowanzellers.com/hellaswag
References in corpus (3)
Cited by in corpus (22)
- Language Models are Few-Shot Learners
- Measuring Massive Multitask Language Understanding
- Natural Adversarial Examples
- Measuring Coding Challenge Competence With APPS
- ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
- Learning from others' mistakes: Avoiding dataset biases without modeling them
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- Abductive Commonsense Reasoning
- HTLM: Hyper-Text Pre-Training and Prompting of Language Models
- A General Language Assistant as a Laboratory for Alignment
- Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
- Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
- Exploring and Analyzing Machine Commonsense Benchmarks
- Knowledge Enhanced Attention for Robust Natural Language Inference
- Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
- An Enhanced Knowledge Injection Model for Commonsense Generation
- AR-LSAT: Investigating Analytical Reasoning of Text
- Counterfactual Story Reasoning and Generation
- Learning and Reasoning for Robot Sequential Decision Making under Uncertainty
- Abductive Reasoning as Self-Supervision for Common Sense Question Answering
- Creative Captioning: An AI Grand Challenge Based on the Dixit Board Game
- Evaluating Document Coherence Modelling