Parallel Attention Network with Sequence Matching for Video Grounding
arXiv:2105.08481 · doi:10.18653/v1/2021.findings-acl.69
Abstract
Given a video, video grounding aims to retrieve a temporal moment that semantically corresponds to a language query. In this work, we propose a Parallel Attention Network with Sequence matching (SeqPAN) to address the challenges in this task: multi-modal representation learning, and target moment boundary prediction. We design a self-guided parallel attention module to effectively capture self-modal contexts and cross-modal attentive information between video and text. Inspired by sequence labeling tasks in natural language processing, we split the ground truth moment into begin, inside, and end regions. We then propose a sequence matching strategy to guide start/end boundary predictions using region labels. Experimental results on three datasets show that SeqPAN is superior to state-of-the-art methods. Furthermore, the effectiveness of the self-guided parallel attention module and the sequence matching module is verified.
arXiv comment15 pages, 10 figures, 7 tables, Findings at ACL 2021
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Bidirectional Attention Flow for Machine Comprehension
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Gradient Estimation Using Stochastic Computation Graphs
- FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
- Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
- ExCL: Extractive Clip Localization Using Natural Language Descriptions
- Tripping through time: Efficient Localization of Activities in Videos