AlignVE: Visual Entailment Recognition Based on Alignment Relations
arXiv:2211.08736 · doi:10.1109/TMM.2022.3222118
Abstract
Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.
arXiv commentThis paper is accepted for publication as a REGULAR paper in the IEEE Transactions on Multimedia
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Enhanced LSTM for Natural Language Inference
- Reasoning about Entailment with Neural Attention
- Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
- Natural Language Inference over Interaction Space
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- Discriminative Neural Sentence Modeling by Tree-Based Convolution
- High-Order Attention Models for Visual Question Answering
- e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations