Modeling Context in Referring Expressions

Abstract

Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets - RefCOCO, RefCOCO+, and RefCOCOg, shows the advantages of our methods for both referring expression generation and comprehension.

19 pages, 6 figures, in ECCV 2016; authors, references and acknowledgement updated

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• LSTM: A Search Space Odyssey
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
• DeePM: A Deep Part-Based Model for Object Detection and Semantic Part Localization

Cited by in corpus (39)

• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Unifying Vision-and-Language Tasks via Text Generation
• ImageNet pre-trained models with batch normalization
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Discriminability objective for training descriptive captions
• Temporal Context Network for Activity Localization in Videos
• TVQA: Localized, Compositional Video Question Answering
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• A Fast and Accurate One-Stage Approach to Visual Grounding
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• GuessWhat?! Visual object discovery through multi-modal dialogue
• Generating Descriptions with Grounded and Co-Referenced People
• Comprehension-guided referring expressions
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
• An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning
• Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• Dense Captioning with Joint Inference and Visual Context
• Spatio-temporal Person Retrieval via Natural Language Queries
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Learning to Disambiguate by Asking Discriminative Questions
• Leveraging Explainability for Comprehending Referring Expressions in the Real World
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Recurrent Instance Segmentation using Sequences of Referring Expressions
• Two-stage Visual Cues Enhancement Network for Referring Image Segmentation
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• Recurrent Multimodal Interaction for Referring Image Segmentation
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories
• Evaluating the Representational Hub of Language and Vision Models
• Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention
• Resolving Referring Expressions in Images With Labeled Elements