Temporal Sentence Grounding in Streaming Videos

Abstract

This paper aims to tackle a novel task - Temporal Sentence Grounding in Streaming Videos (TSGSV). The goal of TSGSV is to evaluate the relevance between a video stream and a given sentence query. Unlike regular videos, streaming videos are acquired continuously from a particular source, and are always desired to be processed on-the-fly in many applications such as surveillance and live-stream analysis. Thus, TSGSV is challenging since it requires the model to infer without future frames and process long historical frames effectively, which is untouched in the early methods. To specifically address the above challenges, we propose two novel methods: (1) a TwinNet structure that enables the model to learn about upcoming events; and (2) a language-guided feature compressor that eliminates redundant visual frames and reinforces the frames that are relevant to the query. We conduct extensive experiments using ActivityNet Captions, TACoS, and MAD datasets. The results demonstrate the superiority of our proposed methods. A systematic ablation study also confirms their effectiveness.

arXiv commentAccepted by ACM MM 2023

References in corpus (7)

• Decoupled Weight Decay Regularization
• Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
• ExCL: Extractive Clip Localization Using Natural Language Descriptions
• HS-GCN: Hamming Spatial Graph Convolutional Networks for Recommendation
• Parallel Attention Network with Sequence Matching for Video Grounding
• Micro-video Tagging via Jointly Modeling Social Influence and Tag Relation
• AntPivot: Livestream Highlight Detection via Hierarchical Attention Mechanism