EVOQUER: Enhancing Temporal Grounding with Video-Pivoted BackQuery Generation

Abstract

Temporal grounding aims to predict a time interval of a video clip corresponding to a natural language query input. In this work, we present EVOQUER, a temporal grounding framework incorporating an existing text-to-video grounding model and a video-assisted query generation network. Given a query and an untrimmed video, the temporal grounding model predicts the target interval, and the predicted video clip is fed into a video translation task by generating a simplified version of the input query. EVOQUER forms closed-loop learning by incorporating loss functions from both temporal grounding and query generation serving as feedback. Our experiments on two widely used datasets, Charades-STA and ActivityNet, show that EVOQUER achieves promising improvements by 1.05 and 1.31 at [email protected]. We also discuss how the query generation task could facilitate error analysis by explaining temporal grounding model behavior.

Accepted by Visually Grounded Interaction and Language (ViGIL) Workshop at NAACL 2021

References in corpus (4)

• Learning Transferable Visual Models From Natural Language Supervision
• ExCL: Extractive Clip Localization Using Natural Language Descriptions
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots