Visuallly Grounded Generation of Entailments from Premises
arXiv:1909.09788
Abstract
Natural Language Inference (NLI) is the task of determining the semantic relationship between a premise and a hypothesis. In this paper, we focus on the {\em generation} of hypotheses from premises in a multimodal setting, to generate a sentence (hypothesis) given an image and/or its description (premise) as the input. The main goals of this paper are (a) to investigate whether it is reasonable to frame NLI as a generation task; and (b) to consider the degree to which grounding textual premises in visual information is beneficial to generation. We compare different neural architectures, showing through automatic and human evaluation that entailments can indeed be generated successfully. We also show that multimodal models outperform unimodal models in this task, albeit marginally.
arXiv commentProceedings of the 12th International Conference on Natural Language Generation (INLG 2019), 11 pages, 5 figures
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Sequence to Sequence Learning with Neural Networks
- Enhanced LSTM for Natural Language Inference
- Reasoning about Entailment with Neural Attention
- Explain Images with Multimodal Recurrent Neural Networks
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- Generating Natural Language Inference Chains
- Pre-gen metrics: Predicting caption quality metrics without generating captions
- Quantifying the amount of visual information used by neural caption generators