CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
arXiv:2104.00285
Abstract
This work concerns video-language pre-training and representation learning. In this now ubiquitous training scheme, a model first performs pre-training on paired videos and text (e.g., video clips and accompanied subtitles) from a large uncurated source corpus, before transferring to specific downstream tasks. This two-stage training process inevitably raises questions about the generalization ability of the pre-trained model, which is particularly pronounced when a salient domain gap exists between source and target data (e.g., instructional cooking videos vs. movies). In this paper, we first bring to light the sensitivity of pre-training objectives (contrastive vs. reconstructive) to domain discrepancy. Then, we propose a simple yet effective framework, CUPID, to bridge this domain gap by filtering and adapting source data to the target data, followed by domain-focused pre-training. Comprehensive experiments demonstrate that pre-training on a considerably small subset of domain-focused data can effectively close the source-target domain gap and achieve significant performance gain, compared to random sampling or even exploiting the full pre-training dataset. CUPID yields new state-of-the-art performance across multiple video-language and video tasks, including text-to-video retrieval [72, 37], video question answering [36], and video captioning [72], with consistent performance lift over different pre-training methods.
arXiv comment12 pages, 4 figures
References in corpus (20)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- A Simple Framework for Contrastive Learning of Visual Representations
- Learning Transferable Visual Models From Natural Language Supervision
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Language Models are Few-Shot Learners
- Unsupervised Learning of Video Representations using LSTMs
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Video (language) modeling: a baseline for generative models of natural videos
- Self-supervised Co-training for Video Representation Learning
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Learning Video Representations using Contrastive Bidirectional Transformer
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Self-Supervised MultiModal Versatile Networks
- How2: A Large-scale Dataset for Multimodal Language Understanding
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
- Spatiotemporal Contrastive Video Representation Learning
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- Multimodal Pretraining for Dense Video Captioning
- MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning