Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
arXiv:2005.14709
Abstract
Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope it will be a useful resource for researchers who propose new datasets, to have a set of tools to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who develop novel architectures, to further understand the implications of their improvements with respect to their model's acquired capabilities.
10 Pages
References in corpus (14)
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
- Learning and Evaluating General Linguistic Intelligence
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- HellaSwag: Can a Machine Really Finish Your Sentence?
- What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
- Compositional Questions Do Not Necessitate Multi-hop Reasoning
- HEAD-QA: A Healthcare Dataset for Complex Reasoning
- Misleading Failures of Partial-input Baselines
- When Choosing Plausible Alternatives, Clever Hans can be Clever
- Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA
- Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
- TWEETQA: A Social Media Focused Question Answering Dataset