CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
arXiv:2106.11097
Abstract
We present CLIP2Video network to transfer the image-language pre-training model to video-text retrieval in an end-to-end manner. Leading approaches in the domain of video-and-language learning try to distill the spatio-temporal video features and multi-modal interaction between videos and languages from a large-scale video-text dataset. Different from them, we leverage pretrained image-language model, simplify it as a two-stage framework with co-learning of image-text and enhancing temporal relations between video frames and video-text respectively, make it able to train on comparatively small datasets. Specifically, based on the spatial semantics captured by Contrastive Language-Image Pretraining (CLIP) model, our model involves a Temporal Difference Block to capture motions at fine temporal video frames, and a Temporal Alignment Block to re-align the tokens of video clips and phrases and enhance the multi-modal correlation. We conduct thorough ablation studies, and achieve state-of-the-art performance on major text-to-video and video-to-text retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT, MSVD and VATEX.
References in corpus (17)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision
- Is Space-Time Attention All You Need for Video Understanding?
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Use What You Have: Video Retrieval Using Representations From Collaborative Experts
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- An Image is Worth 16x16 Words, What is a Video Worth?
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
- Support-set bottlenecks for video-text representation learning
- Learning Spatiotemporal Features via Video and Text Pair Discrimination
- Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
- Learning Video Representations from Textual Web Supervision
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval
- Cross-Modal and Hierarchical Modeling of Video and Text
Cited by in corpus (17)
- VLP: A Survey on Vision-Language Pre-training
- ActionCLIP: A New Paradigm for Video Action Recognition
- Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
- Cross-Modal Adapter for Vision-Language Retrieval
- CoVR-2: Automatic Data Construction for Composed Video Retrieval
- Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning
- Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language
- Learn to Understand Negation in Video Retrieval
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
- IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
- A CLIP-Enhanced Method for Video-Language Understanding
- STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
- An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval
- SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
- WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge
- CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
- An implementation of the "Guess who?" game using CLIP