DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
arXiv:2010.06260
Abstract
This paper studies the task of temporal moment localization in a long untrimmed video using natural language query. Given a query sentence, the goal is to determine the start and end of the relevant segment within the video. Our key innovation is to learn a video feature embedding through a language-conditioned message-passing algorithm suitable for temporal moment localization which captures the relationships between humans, objects and activities in the video. These relationships are obtained by a spatial sub-graph that contextualizes the scene representation using detected objects and human features conditioned in the language query. Moreover, a temporal sub-graph captures the activities within the video through time. Our method is evaluated on three standard benchmark datasets, and we also introduce YouCookII as a new benchmark for this task. Experiments show our method outperforms state-of-the-art methods on these datasets, confirming the effectiveness of our approach.
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Coherent Multi-Sentence Video Description with Variable Level of Detail
- ExCL: Extractive Clip Localization Using Natural Language Descriptions
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals