Learning Spatiotemporal Features via Video and Text Pair Discrimination
arXiv:2001.05691
Abstract
Current video representations heavily rely on learning from manually annotated video datasets which are time-consuming and expensive to acquire. We observe videos are naturally accompanied by abundant text information such as YouTube titles and Instagram captions. In this paper, we leverage this visual-textual connection to learn spatiotemporal features in an efficient weakly-supervised manner. We present a general cross-modal pair discrimination (CPD) framework to capture this correlation between a video and its associated text. Specifically, we adopt noise-contrastive estimation to tackle the computational issue imposed by the huge amount of pair instance classes and design a practical curriculum learning strategy. We train our CPD models on both standard video dataset (Kinetics-210k) and uncurated web video dataset (Instagram-300k) to demonstrate its effectiveness. Without further fine-tuning, the learnt models obtain competitive results for action classification on Kinetics under the linear classification protocol. Moreover, our visual model provides an effective initialization to fine-tune on downstream tasks, which yields a remarkable performance gain for action recognition on UCF101 and HMDB51, compared with the existing state-of-the-art self-supervised training methods. In addition, our CPD model yields a new state of the art for zero-shot action recognition on UCF101 by directly utilizing the learnt visual-textual embeddings. The code will be made available at https://github.com/MCG-NJU/CPD-Video.
arXiv commentTechnical Report
References in corpus (20)
- Efficient Estimation of Word Representations in Vector Space
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distilling the Knowledge in a Neural Network
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Learning deep representations by mutual information estimation and maximization
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Contrastive Multiview Coding
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
- Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- VideoBERT: A Joint Model for Video and Language Representation Learning
- SlowFast Networks for Video Recognition
- Long-term Temporal Convolutions for Action Recognition
- Video Representation Learning by Dense Predictive Coding
- Omni-sourced Webly-supervised Learning for Video Recognition
- Learning Video Representations from Textual Web Supervision
Cited by in corpus (9)
- What Makes for Good Views for Contrastive Learning?
- CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
- Labelling unlabelled videos from scratch with multi-modal self-supervision
- Video Understanding as Machine Translation
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
- Learning Video Representations from Textual Web Supervision
- Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
- Boosting Video Representation Learning with Multi-Faceted Integration
- On Compositions of Transformations in Contrastive Self-Supervised Learning