RTQ: Rethinking Video-language Understanding Based on Image-text Model
arXiv:2312.00347 · doi:10.1145/3581783.3612152
Abstract
Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos. However, video-language understanding presents unique challenges due to the inclusion of highly complex semantic details, which result in information redundancy, temporal dependency, and scene complexity. Current techniques have only partially tackled these issues, and our quantitative analysis indicates that some of these methods are complementary. In light of this, we propose a novel framework called RTQ (Refine, Temporal model, and Query), which addresses these challenges simultaneously. The approach involves refining redundant information within frames, modeling temporal relations among frames, and querying task-specific information from the videos. Remarkably, our model demonstrates outstanding performance even in the absence of video-language pre-training, and the results are comparable with or superior to those achieved by state-of-the-art pre-training methods. Code is available at https://github.com/SCZwangxiao/RTQ-MM2023.
arXiv commentAccepted by ACM MM 2023 as Oral representation
References in corpus (18)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Flamingo: a Visual Language Model for Few-Shot Learning
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
- CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
- OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
- Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
- CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
- mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
- VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
- Multi-queue Momentum Contrast for Microvideo-Product Retrieval
- Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
- HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
- Hierarchical Modular Network for Video Captioning
- Micro-video Tagging via Jointly Modeling Social Influence and Tag Relation
- STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training