When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs

Abstract

We consider the task of temporal human action localization in lifestyle vlogs. We introduce a novel dataset consisting of manual annotations of temporal localization for 13,000 narrated actions in 1,200 video clips. We present an extensive analysis of this data, which allows us to better understand how the language and visual modalities interact throughout the videos. We propose a simple yet effective method to localize the narrated actions based on their expected duration. Through several experiments and analyses, we show that our method brings complementary information with respect to previous methods, and leads to improvements over previous work for the task of temporal action localization.

arXiv commentarXiv admin note: text overlap with arXiv:1906.04236

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• The Kinetics Human Action Video Dataset
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Single Shot Temporal Action Detection
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Stacked Attention Networks for Image Question Answering
• Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
• ExCL: Extractive Clip Localization Using Natural Language Descriptions
• Identifying Visible Actions in Lifestyle Vlogs
• End-to-End Learning of Visual Representations from Uncurated Instructional Videos