STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
arXiv:2302.09736 · doi:10.1609/aaai.v37i3.25483
Abstract
Although large-scale video-language pre-training models, which usually build a global alignment between the video and the text, have achieved remarkable progress on various downstream tasks, the idea of adopting fine-grained information during the pre-training stage is not well explored. In this work, we propose STOA-VLP, a pre-training framework that jointly models object and action information across spatial and temporal dimensions. More specifically, the model regards object trajectories across frames and multiple action features from the video as fine-grained features. Besides, We design two auxiliary tasks to better incorporate both kinds of information into the pre-training process of the video-language model. The first is the dynamic object-text alignment task, which builds a better connection between object trajectories and the relevant noun tokens. The second is the spatial-temporal action set prediction, which guides the model to generate consistent action features by predicting actions found in the text. Extensive experiments on three downstream tasks (video captioning, text-video retrieval, and video question answering) demonstrate the effectiveness of our proposed STOA-VLP (e.g. 3.7 Rouge-L improvements on MSR-VTT video captioning benchmark, 2.9% accuracy improvements on MSVD video question answering benchmark, compared to previous approaches).
arXiv commentAAAI 2023, 7 pages, 3 figures
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Learning Video Representations using Contrastive Bidirectional Transformer
- UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
- VideoBERT: A Joint Model for Video and Language Representation Learning
- CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
- On the Relationship between Self-Attention and Convolutional Layers
- Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
- VinVL: Revisiting Visual Representations in Vision-Language Models
- MERLOT: Multimodal Neural Script Knowledge Models
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Object Relational Graph with Teacher-Recommended Learning for Video Captioning
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
- Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
- Align and Prompt: Video-and-Language Pre-training with Entity Prompts
- CLIP2TV: Align, Match and Distill for Video-Text Retrieval
- End-to-end Generative Pretraining for Multimodal Video Captioning
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
- Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
- VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding