e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations

Abstract

The recently proposed SNLI-VE corpus for recognising visual-textual entailment is a large, real-world dataset for fine-grained multimodal reasoning. However, the automatic way in which SNLI-VE has been assembled (via combining parts of two related datasets) gives rise to a large number of errors in the labels of this corpus. In this paper, we first present a data collection effort to correct the class with the highest error rate in SNLI-VE. Secondly, we re-evaluate an existing model on the corrected corpus, which we call SNLI-VE-2.0, and provide a quantitative comparison with its performance on the non-corrected corpus. Thirdly, we introduce e-SNLI-VE, which appends human-written natural language explanations to SNLI-VE-2.0. Finally, we train models that learn from these explanations at training time, and output such explanations at testing time.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• e-SNLI: Natural Language Inference with Natural Language Explanations
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Multimodal Logical Inference System for Visual-Textual Entailment

Cited by in corpus (7)

• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Explaining Deep Neural Networks
• CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• AlignVE: Visual Entailment Recognition Based on Alignment Relations
• Black-box Attacks on Image Activity Prediction and its Natural Language Explanations
• Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference