Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Abstract

Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset. In this paper, we introduce a new inference task, Visual Entailment (VE) - consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at https://github.com/ necla-ml/SNLI-VE.

References in corpus (35)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Going Deeper with Convolutions
• Enhanced LSTM for Natural Language Inference
• VQA: Visual Question Answering
• Reasoning about Entailment with Neural Attention
• A simple neural network module for relational reasoning
• A large annotated corpus for learning natural language inference
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• To Trust Or Not To Trust A Classifier
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Compositional Attention Networks for Machine Reasoning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Scene Graph Generation by Iterative Message Passing
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Bilinear Attention Networks
• Detecting Visual Relationships with Deep Relational Networks
• Visual Translation Embedding Network for Visual Relation Detection
• Detecting and Recognizing Human-Object Interactions
• Attentive Explanations: Justifying Decisions and Pointing to the Evidence
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Compact Bilinear Pooling
• Visual7W: Grounded Question Answering in Images
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
• Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
• Learning to Detect Human-Object Interactions
• Attention on Attention: Architectures for Visual Question Answering (VQA)
• Modeling Relationships in Referential Expressions with Compositional Modular Networks

Cited by in corpus (35)

• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• VLP: A Survey on Vision-Language Pre-training
• UNITER: UNiversal Image-TExt Representation Learning
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• A Review of Emerging Research Directions in Abstract Visual Reasoning
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Image Retrieval from Contextual Descriptions
• CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Visual Reasoning: from State to Transformation
• AlignVE: Visual Entailment Recognition Based on Alignment Relations
• Check It Again: Progressive Visual Question Answering via Visual Entailment
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Evaluating Multimodal Representations on Visual Semantic Textual Similarity
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Transformation Driven Visual Reasoning
• TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning
• Visuallly Grounded Generation of Entailments from Premises
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding