MAttNet: Modular Attention Network for Referring Expression Comprehension

Abstract

In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: language-based attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components. Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks. Demo and code are provided.

Equation of word attention fixed; MAttNet+Grabcut results added

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Implementation of Faster RCNN with Study for Region Sampling
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Modeling Context in Referring Expressions
• Boosting Image Captioning with Attributes
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Cited by in corpus (21)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Unifying Vision-and-Language Tasks via Text Generation
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• TVQA: Localized, Compositional Video Question Answering
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• A Fast and Accurate One-Stage Approach to Visual Grounding
• A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension
• Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding
• Referring Expression Object Segmentation with Caption-Aware Consistency
• Exploiting Temporal Relationships in Video Moment Localization with Natural Language
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Reflective Decoding Network for Image Captioning
• ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
• Referring Image Segmentation via Cross-Modal Progressive Comprehension
• Leveraging Explainability for Comprehending Referring Expressions in the Real World
• Modularized Textual Grounding for Counterfactual Resilience
• Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attention
• Recurrent Instance Segmentation using Sequences of Referring Expressions
• Sentiment Tagging with Partial Labels using Modular Architectures
• Generating Easy-to-Understand Referring Expressions for Target Identifications