Tripping through time: Efficient Localization of Activities in Videos

Abstract

Localizing moments in untrimmed videos via language queries is a new and interesting task that requires the ability to accurately ground language into video. Previous works have approached this task by processing the entire video, often more than once, to localize relevant activities. In the real world applications of this approach, such as video surveillance, efficiency is a key system requirement. In this paper, we present TripNet, an end-to-end system that uses a gated attention architecture to model fine-grained textual and visual representations in order to align text and video content. Furthermore, TripNet uses reinforcement learning to efficiently localize relevant activity clips in long videos, by learning how to intelligently skip around the video. It extracts visual features for few frames to perform activity classification. In our evaluation over Charades-STA, ActivityNet Captions and the TACoS dataset, we find that TripNet achieves high accuracy and saves processing time by only looking at 32-41% of the entire video.

arXiv commentPresented at BMVC, 2020

References in corpus (4)

• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Temporal Context Network for Activity Localization in Videos
• Weakly-Supervised Alignment of Video With Text
• Unsupervised Semantic Parsing of Video Collections

Cited by in corpus (8)

• Temporal Sentence Grounding in Videos: A Survey and Future Directions
• Parallel Attention Network with Sequence Matching for Video Grounding
• End-to-end Multi-modal Video Temporal Grounding
• Progressive Localization Networks for Language-based Moment Localization
• Hierarchical Deep Residual Reasoning for Temporal Moment Localization
• Towards Debiasing Temporal Sentence Grounding in Video
• VLG-Net: Video-Language Graph Matching Network for Video Grounding
• A Survey on Temporal Sentence Grounding in Videos