Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

Abstract

Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity. To overcome this limitation, we present a method for generating data sets of multiply-quantified natural language inference (NLI) examples in which semantic complexity can be precisely characterized, and we use this method to show that a variety of common architectures for NLI inevitably fail to encode crucial information; only a model with forced lexical alignments avoids this damaging information loss.

References in corpus (2)

• Adam: A Method for Stochastic Optimization
• Reasoning about Entailment with Neural Attention

Cited by in corpus (7)

• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• ANLIzing the Adversarial Natural Language Inference Dataset
• HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning
• An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference
• Posing Fair Generalization Tasks for Natural Language Inference
• Logic, Language, and Calculus