Video-Text Pre-training with Learned Regions
arXiv:2112.01194
Abstract
Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information. State-of-the-art approaches extract visual features from raw pixels in an end-to-end fashion. However, these methods operate at frame-level directly and thus overlook the spatio-temporal structure of objects in video, which yet has a strong synergy with nouns in textual descriptions. In this work, we propose a simple yet effective module for video-text representation learning, namely RegionLearner, which can take into account the structure of objects during pre-training on large-scale video-text pairs. Given a video, our module (1) first quantizes visual features into semantic clusters, then (2) generates learnable masks and uses them to aggregate the features belonging to the same semantic region, and finally (3) models the interactions between different aggregated regions. In contrast to using off-the-shelf object detectors, our proposed module does not require explicit supervision and is much more computationally efficient. We pre-train the proposed approach on the public WebVid2M and CC3M datasets. Extensive evaluations on four downstream video-text retrieval benchmarks clearly demonstrate the effectiveness of our RegionLearner. The code will be available at https://github.com/ruiyan1995/Region_Learner.
References in corpus (17)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Learning Transferable Visual Models From Natural Language Supervision
- The Kinetics Human Action Video Dataset
- Neural Discrete Representation Learning
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Is Space-Time Attention All You Need for Video Understanding?
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- Video Swin Transformer
- Video Understanding as Machine Translation
- TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
- Interactive Fusion of Multi-level Features for Compositional Activity Recognition