Video Moment Retrieval with Text Query Considering Many-to-Many Correspondence Using Potentially Relevant Pair
arXiv:2106.13566
Abstract
In this paper we undertake the task of text-based video moment retrieval from a corpus of videos. To train the model, text-moment paired datasets were used to learn the correct correspondences. In typical training methods, ground-truth text-moment pairs are used as positive pairs, whereas other pairs are regarded as negative pairs. However, aside from the ground-truth pairs, some text-moment pairs should be regarded as positive. In this case, one text annotation can be positive for many video moments. Conversely, one video moment can be corresponded to many text annotations. Thus, there are many-to-many correspondences between the text annotations and video moments. Based on these correspondences, we can form potentially relevant pairs, which are not given as ground truth yet are not negative; effectively incorporating such relevant pairs into training can improve the retrieval performance. The text query should describe what is happening in a video moment. Hence, different video moments annotated with similar texts, which contain a similar action, are likely to hold the similar action, thus these pairs can be considered as potentially relevant pairs. In this paper, we propose a novel training method that takes advantage of potentially relevant pairs, which are detected based on linguistic analysis about text annotation. Experiments on two benchmark datasets revealed that our method improves the retrieval performance both quantitatively and qualitatively.
arXiv comment15 pages, 10 figures
References in corpus (10)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- TALL: Temporal Activity Localization via Language Query
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Constituency Parsing with a Self-Attentive Encoder
- Text-based Localization of Moments in a Video Corpus
- A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval
- The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection