Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
arXiv:1805.02834
Abstract
We study weakly-supervised video object grounding: given a video segment and a corresponding descriptive sentence, the goal is to localize objects that are mentioned from the sentence in the video. During training, no object bounding boxes are available, but the set of possible objects to be grounded is known beforehand. Existing approaches in the image domain use Multiple Instance Learning (MIL) to ground objects by enforcing matches between visual and semantic features. A naive extension of this approach to the video domain is to treat the entire segment as a bag of spatial object proposals. However, an object existing sparsely across multiple frames might not be detected completely since successfully spotting it from one single frame would trigger a satisfactory match. To this end, we propagate the weak supervisory signal from the segment level to frames that likely contain the target object. For frames that are unlikely to contain the target objects, we use an alternative penalty loss. We also leverage the interactions among objects as a textual guide for the grounding. We evaluate our model on the newly-collected benchmark YouCook2-BoundingBox and show improvements over competitive baselines.
arXiv comment16 pages including Appendix
References in corpus (11)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Towards Automatic Learning of Procedures from Web Instructional Videos
- MAttNet: Modular Attention Network for Referring Expression Comprehension
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- End-to-End Dense Video Captioning with Masked Transformer
- Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
- Generating Descriptions with Grounded and Co-Referenced People
- Unsupervised Object Discovery and Tracking in Video Collections
- Attend and Interact: Higher-Order Object Interactions for Video Understanding
Cited by in corpus (20)
- Learning Video Representations using Contrastive Bidirectional Transformer
- VideoBERT: A Joint Model for Video and Language Representation Learning
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
- Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
- Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
- Explainable Goal-Driven Agents and Robots -- A Comprehensive Review
- Grounding Object Detections With Transcriptions
- Activity Driven Weakly Supervised Object Detection
- A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
- Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
- Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images
- COBE: Contextualized Object Embeddings from Narrated Instructional Video
- Grounded Video Description
- DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
- Co-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videos
- Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding
- Designing Multimodal Datasets for NLP Challenges
- Visual Relation Grounding in Videos
- ViP: Video Platform for PyTorch