Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries

Abstract

Recognising objects according to a pre-defined fixed set of class labels has been well studied in the Computer Vision. There are a great many practical applications where the subjects that may be of interest are not known beforehand, or so easily delineated, however. In many of these cases natural language dialog is a natural way to specify the subject of interest, and the task achieving this capability (a.k.a, Referring Expression Comprehension) has recently attracted attention. To this end we propose a unified framework, the ParalleL AttentioN (PLAN) network, to discover the object in an image that is being referred to in variable length natural expression descriptions, from short phrases query to long multi-round dialogs. The PLAN network has two attention mechanisms that relate parts of the expressions to both the global visual content and also directly to object candidates. Furthermore, the attention mechanisms are recurrent, making the referring process visualizable and explainable. The attended information from these dual sources are combined to reason about the referred object. These two attention mechanisms can be trained in parallel and we find the combined system outperforms the state-of-art on several benchmarked datasets with different length language input, such as RefCOCO, RefCOCO+ and GuessWhat?!.

11 pages

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Image Captioning with Semantic Attention
• Show and Tell: A Neural Image Caption Generator
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Generation and Comprehension of Unambiguous Object Descriptions
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Review Networks for Caption Generation
• Modeling Context in Referring Expressions
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Structured Attentions for Visual Question Answering
• Natural Language Object Retrieval
• When Unsupervised Domain Adaptation Meets Tensor Representations
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• An Analysis of Visual Question Answering Algorithms
• Modeling Relationships in Referential Expressions with Compositional Modular Networks
• Comprehension-guided referring expressions
• Learning a Deep Embedding Model for Zero-Shot Learning
• An Empirical Study of Language CNN for Image Captioning