Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
arXiv:2005.14709
Abstract
Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope it will be a useful resource for researchers who propose new datasets, to have a set of tools to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who develop novel architectures, to further understand the implications of their improvements with respect to their model's acquired capabilities.
arXiv comment10 Pages
References in corpus (38)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Adversarially Robust Generalization Requires More Data
- A large annotated corpus for learning natural language inference
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- Learning and Evaluating General Linguistic Intelligence
- A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
- Transforming Question Answering Datasets Into Natural Language Inference Datasets
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Adversarial NLI: A New Benchmark for Natural Language Understanding
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
- Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
- HellaSwag: Can a Machine Really Finish Your Sentence?
- Stress Test Evaluation for Natural Language Inference
- Probing Neural Network Comprehension of Natural Language Arguments
- What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
- Robust Machine Comprehension Models via Adversarial Training
- Compositional Questions Do Not Necessitate Multi-hop Reasoning
- emrQA: A Large Corpus for Question Answering on Electronic Medical Records
- AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples
- WIQA: A dataset for "What if..." reasoning over procedural text
- Reasoning Over Paragraph Effects in Situations
- HEAD-QA: A Healthcare Dataset for Complex Reasoning
- Misleading Failures of Partial-input Baselines
- When Choosing Plausible Alternatives, Clever Hans can be Clever
- A Framework for Evaluation of Machine Reading Comprehension Gold Standards
- Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
- Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA
- Posing Fair Generalization Tasks for Natural Language Inference
- TWEETQA: A Social Media Focused Question Answering Dataset
- Does it care what you asked? Understanding Importance of Verbs in Deep Learning QA System