Comprehension-guided referring expressions

Abstract

We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic "image captioning" which lacks natural standard evaluation criteria, quality of a referring expression may be measured by the receiver's ability to correctly infer which object is being described. Following this intuition, we propose two approaches to utilize models trained for comprehension task to generate better expressions. First, we use a comprehension module trained on human-generated expressions, as a "critic" of referring expression generator. The comprehension module serves as a differentiable proxy of human evaluation, providing training signal to the generation module. Second, we use the comprehension module in a generate-and-rerank pipeline, which chooses from candidate expressions generated by a model according to their performance on the comprehension task. We show that both approaches lead to improved referring expression generation on multiple benchmark datasets.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• An Actor-Critic Algorithm for Sequence Prediction
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Modeling Context in Referring Expressions

Cited by in corpus (9)

• Discriminability objective for training descriptive captions
• Learning to Assemble Neural Module Tree Networks for Visual Grounding
• A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension
• Learning to Globally Edit Images with Textual Description
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• Unified Pragmatic Models for Generating and Following Instructions
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Generating Easy-to-Understand Referring Expressions for Target Identifications
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries