MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

Abstract

Given a query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query. In this paper, we adopt a proposal-based solution that generates proposals (i.e., candidate moments) and then select the best matching proposal. On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling. The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR (DEtection TRansformer) framework. To achieve this, we design a multi-scale visual-linguistic encoder, and an anchor-guided moment decoder paired with a set of learnable templates. Experimental results on three public datasets demonstrate the superior performance of MS-DETR.

arXiv commentAccepted by ACL 2023

References in corpus (9)

• DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
• DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
• Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
• Parallel Attention Network with Sequence Matching for Video Grounding
• Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment
• Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding
• Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language