Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators

Abstract

Grounding language to visual relations is critical to various language-and-vision applications. In this work, we tackle two fundamental language-and-vision tasks: image-text matching and image captioning, and demonstrate that neural scene graph generators can learn effective visual relation features to facilitate grounding language to visual relations and subsequently improve the two end applications. By combining relation features with the state-of-the-art models, our experiments show significant improvement on the standard Flickr30K and MSCOCO benchmarks. Our experimental results and analysis show that relation features improve downstream models' capability of capturing visual relations in end vision-and-language applications. We also demonstrate the importance of learning scene graph generators with visually relevant relations to the effectiveness of relation features.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Attention-Based Models for Speech Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Pixels to Graphs by Associative Embedding
• From Captions to Visual Concepts and Back
• CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• Relational Reasoning using Prior Knowledge for Visual Captioning
• VrR-VG: Refocusing Visually-Relevant Relationships

Cited by in corpus (6)

• A Comprehensive Survey of Scene Graphs: Generation and Application
• Deep Learning Approaches on Image Captioning: A Review
• Plug-and-Play Regulators for Image-Text Matching
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Target-Oriented Deformation of Visual-Semantic Embedding Space
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language