Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
arXiv:1705.01371
Abstract
We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i.e., localize) arbitrary linguistic phrases, in the form of spatial attention masks. Specifically, the model is trained with images and their associated image-level captions, without any explicit region-to-phrase correspondence annotations. To this end, we introduce an end-to-end model which learns visual groundings of phrases with two types of carefully designed loss functions. In addition to the standard discriminative loss, which enforces that attended image regions and phrases are consistently encoded, we propose a novel structural loss which makes use of the parse tree structures induced by the sentences. In particular, we ensure complementarity among the attention masks that correspond to sibling noun phrases, and compositionality of attention masks among the children and parent phrases, as defined by the sentence parse tree. We validate the effectiveness of our approach on the Microsoft COCO and Visual Genome datasets.
arXiv commentCVPR 2017
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Fully Convolutional Networks for Semantic Segmentation
- Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
- Simple Baseline for Visual Question Answering
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Learning Deep Features for Discriminative Localization
- ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
- Segmentation from Natural Language Expressions
- Natural Language Object Retrieval
- End-to-End Localization and Ranking for Relative Attributes
Cited by in corpus (10)
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos
- REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
- Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
- Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
- Sub-Instruction Aware Vision-and-Language Navigation
- Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
- Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
- MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
- Self-view Grounding Given a Narrated 360° Video