Visual Entailment Task for Visually-Grounded Language Learning

Abstract

We introduce a new inference task - Visual Entailment (VE) - which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE (publicly available at https://github.com/necla-ml/SNLI-VE) is proposed for VE tasks based on the Stanford Natural Language Inference corpus and Flickr30k. We introduce a differentiable architecture called the Explainable Visual Entailment model (EVE) to tackle the VE problem. EVE and several other state-of-the-art visual question answering (VQA) based models are evaluated on the SNLI-VE dataset, facilitating grounded language understanding and providing insights on how modern VQA based models perform.

arXiv comment4 pages, accepted by Visually Grounded Interaction and Language (ViGIL) workshop in NeurIPS 2018

References in corpus (7)

• VQA: Visual Question Answering
• A simple neural network module for relational reasoning
• A large annotated corpus for learning natural language inference
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Compositional Attention Networks for Machine Reasoning
• Bilinear Attention Networks
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Cited by in corpus (7)

• VLP: A Survey on Vision-Language Pre-training
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• Multimodal Logical Inference System for Visual-Textual Entailment
• Transformation Driven Visual Reasoning