Adversarial NLI: A New Benchmark for Natural Language Understanding

Abstract

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

arXiv commentACL 2020

References in corpus (23)

• VQA: Visual Question Answering
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• A large annotated corpus for learning natural language inference
• Synthetic and Natural Noise Both Break Neural Machine Translation
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
• Natural Language Inference over Interaction Space
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Exploring Nearest Neighbor Approaches for Image Captioning
• Adversarial Filters of Dataset Biases
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
• No Training Required: Exploring Random Encoders for Sentence Classification
• Multi-column Deep Neural Networks for Image Classification
• Abductive Commonsense Reasoning
• Teaching Machines to Describe Images via Natural Language Feedback
• Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
• Build It, Break It, Fix It: Contesting Secure Development
• CODAH: An Adversarially Authored Question-Answer Dataset for Common Sense

Cited by in corpus (25)

• Language Models are Few-Shot Learners
• Deep Learning Based Text Classification: A Comprehensive Review
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Recipes for Safety in Open-domain Chatbots
• Adversarial Training for Large Neural Language Models
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Syntactic Data Augmentation Increases Robustness to Inference Heuristics
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding
• Adversarial NLI for Factual Correctness in Text Summarisation Models
• Improving Classification through Weak Supervision in Context-specific Conversational Agent Development for Teacher Education
• Deploying Lifelong Open-Domain Dialogue Learning
• RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
• AR-LSAT: Investigating Analytical Reasoning of Text
• DQI: A Guide to Benchmark Evaluation
• Our Evaluation Metric Needs an Update to Encourage Generalization
• Generating Semantically Valid Adversarial Questions for TableQA
• Data vs classifiers, who wins?
• Geometry matters: Exploring language examples at the decision boundary
• Logic, Language, and Calculus
• Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver