Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding
arXiv:2210.11933 · doi:10.18653/v1/2021.findings-emnlp.9
Abstract
Temporal language grounding (TLG) aims to localize a video segment in an untrimmed video based on a natural language description. To alleviate the expensive cost of manual annotations for temporal boundary labels, we are dedicated to the weakly supervised setting, where only video-level descriptions are provided for training. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. However, the temporal structure of the video as well as the complicated semantics in the sentence are lost during the learning. In this work, we propose a novel candidate-free framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG. Instead of view the sentence and candidate moments as a whole, FSAN learns token-by-clip cross-modal semantic alignment by an iterative cross-modal interaction module, generates a fine-grained cross-modal semantic alignment map, and performs grounding directly on top of the map. Extensive experiments are conducted on two widely-used benchmarks: ActivityNet-Captions, and DiDeMo, where our FSAN achieves state-of-the-art performance.
arXiv comment11 pages, 4 figures, accepted by Findings of EMNLP 2021
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
- Pre-Trained Image Processing Transformer
- VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval
- Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
- DeVLBert: Learning Deconfounded Visio-Linguistic Representations