Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators

Abstract

Grounding language to visual relations is critical to various language-and-vision applications. In this work, we tackle two fundamental language-and-vision tasks: image-text matching and image captioning, and demonstrate that neural scene graph generators can learn effective visual relation features to facilitate grounding language to visual relations and subsequently improve the two end applications. By combining relation features with the state-of-the-art models, our experiments show significant improvement on the standard Flickr30K and MSCOCO benchmarks. Our experimental results and analysis show that relation features improve downstream models' capability of capturing visual relations in end vision-and-language applications. We also demonstrate the importance of learning scene graph generators with visually relevant relations to the effectiveness of relation features.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Attention-Based Models for Speech Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Relational Reasoning using Prior Knowledge for Visual Captioning

Cited by in corpus (6)

• A Comprehensive Survey of Scene Graphs: Generation and Application
• Deep Learning Approaches on Image Captioning: A Review
• Plug-and-Play Regulators for Image-Text Matching
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Target-Oriented Deformation of Visual-Semantic Embedding Space
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language