Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Abstract

In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatio-temporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their matching behaviors. Besides a ranking loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unreliable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the ImageNet video object detection dataset, to serve as a benchmark for our task. Extensive experimental results demonstrate the superiority of our model over the baseline approaches.

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Joint Model of Language and Perception for Grounded Attribute Learning
• Using Syntax to Ground Referring Expressions in Natural Images
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• Generating Descriptions with Grounded and Co-Referenced People
• AMC: Attention guided Multi-modal Correlation Learning for Image Search
• Sentence Directed Video Object Codetection

Cited by in corpus (5)

• Text-based Localization of Moments in a Video Corpus
• Human-centric Spatio-Temporal Video Grounding With Visual Transformers
• Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
• Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval