MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment

Abstract

This research strives for natural language moment retrieval in long, untrimmed video streams. The problem is not trivial especially when a video contains multiple moments of interests and the language describes complex temporal dependencies, which often happens in real scenarios. We identify two crucial challenges: semantic misalignment and structural misalignment. However, existing approaches treat different moments separately and do not explicitly model complex moment-wise temporal relations. In this paper, we present Moment Alignment Network (MAN), a novel framework that unifies the candidate moment encoding and temporal structural reasoning in a single-shot feed-forward network. MAN naturally assigns candidate moment representations aligned with language semantics over different temporal locations and scales. Most importantly, we propose to explicitly model moment-wise temporal relations as a structured graph and devise an iterative graph adjustment network to jointly learn the best structure in an end-to-end manner. We evaluate the proposed approach on two challenging public benchmarks DiDeMo and Charades-STA, where our MAN significantly outperforms the state-of-the-art by a large margin.

arXiv commentCVPR 2019

References in corpus (6)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning

Cited by in corpus (5)

• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• WSLLN: Weakly Supervised Natural Language Localization Networks
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Compositional Temporal Visual Grounding of Natural Language Event Descriptions
• Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos