Modeling Relationships in Referential Expressions with Compositional Modular Networks

Abstract

People often refer to entities in an image in terms of their relationships with other entities. For example, "the black cat sitting under the table" refers to both a "black cat" entity and its relationship with another "table" entity. Understanding these relationships is essential for interpreting and grounding such natural language expressions. Most prior work focuses on either grounding entire referential expressions holistically to one region, or localizing relationships based on a fixed set of categories. In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene. We call this approach Compositional Modular Networks (CMNs): a novel architecture that learns linguistic analysis and visual inference end-to-end. Our approach is built around two types of neural modules that inspect local regions and pairwise interactions between regions. We evaluate CMNs on multiple referential expression datasets, outperforming state-of-the-art approaches on all tasks.

References in corpus (1)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Cited by in corpus (14)

• Pixels to Graphs by Associative Embedding
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• Grounding Referring Expressions in Images by Variational Context
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding
• Grounded Objects and Interactions for Video Captioning
• Embodied Language Grounding with 3D Visual Feature Representations
• Learning Actor Relation Graphs for Group Activity Recognition
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries