Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
arXiv:1805.02266
Abstract
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set. Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
arXiv comment6 pages, short paper at ACL 2018
References in corpus (7)
- Enhanced LSTM for Natural Language Inference
- Reasoning about Entailment with Neural Attention
- Neural Natural Language Inference Models Enhanced with External Knowledge
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Natural Language Inference over Interaction Space
- Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
- Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
Cited by in corpus (7)
- Shortcut Learning in Deep Neural Networks
- BERT-ATTACK: Adversarial Attack Against BERT Using BERT
- Improving Natural Language Inference Using External Knowledge in the Science Questions Domain
- Knowledge Enhanced Attention for Robust Natural Language Inference
- Path-Based Contextualization of Knowledge Graphs for Textual Entailment
- Geometry matters: Exploring language examples at the decision boundary
- Unsupervised Pre-training with Structured Knowledge for Improving Natural Language Inference