Learning Video Representations from Textual Web Supervision

Abstract

Videos on the Internet are paired with pieces of text, such as titles and descriptions. This text typically describes the most important content in the video, such as the objects in the scene and the actions being performed. Based on this observation, we propose to use text as a method for learning video representations. To accomplish this, we propose a data collection process and use it to collect 70M video clips shared publicly on the Internet, and we then train a model to pair each video with its associated text. We evaluate the model on several down-stream action recognition tasks, including Kinetics, HMDB-51, and UCF-101. We find that this approach is an effective method of pre-training video representations. Specifically, it outperforms all existing methods for self-supervised and cross-modal video representation learning.

References in corpus (20)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• SGDR: Stochastic Gradient Descent with Warm Restarts
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Contrastive Multiview Coding
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• A Short Note about Kinetics-600
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Spatiotemporal Contrastive Video Representation Learning
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Learning for Video Correspondence Flow
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data

Cited by in corpus (5)

• Learning Transferable Visual Models From Natural Language Supervision
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Revisiting 3D ResNets for Video Recognition
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning