Spatiotemporal Contrastive Video Representation Learning

Abstract

We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial. We carefully design data augmentations involving spatial and temporal cues. Concretely, we propose a temporally consistent spatial augmentation method to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. We also propose a sampling-based temporal augmentation method to avoid overly enforcing invariance on clips that are distant in time. On Kinetics-600, a linear classifier trained on the representations learned by CVRL achieves 70.4% top-1 accuracy with a 3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated R3D-50. The performance of CVRL can be further improved to 72.9% with a larger R3D-152 (2x filters) backbone, significantly closing the gap between unsupervised and supervised video representation learning. Our code and models will be available at https://github.com/tensorflow/models/tree/master/official/.

arXiv commentCVPR2021 Camera ready

References in corpus (24)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Learning of Video Representations using LSTMs
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Large Batch Training of Convolutional Networks
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• A Short Note about Kinetics-600
• Self-supervised Co-training for Video Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Video Representation Learning by Pace Prediction
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Can Temporal Information Help with Contrastive Self-Supervised Learning?

Cited by in corpus (29)

• Self-supervised Co-training for Video Representation Learning
• Revisiting ResNets: Improved Training and Scaling Strategies
• Self-Supervised MultiModal Versatile Networks
• TCLR: Temporal Contrastive Learning for Video Representation
• Self-Supervised Learning for Videos: A Survey
• Semantics-Guided Contrastive Network for Zero-Shot Object detection
• A Comprehensive Study of Deep Video Action Recognition
• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
• Semi-TCL: Semi-Supervised Track Contrastive Representation Learning
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion
• Learning Video Representations from Textual Web Supervision
• CLAR: Contrastive Learning of Auditory Representations
• Hierarchical Contrastive Motion Learning for Video Action Recognition
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Contrastive Learning for Robust Android Malware Familial Classification
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• Motion-Focused Contrastive Learning of Video Representations
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Long-Short Temporal Contrastive Learning of Video Transformers
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Spatio-Temporal Video Representation Learning for AI Based Video Playback Style Prediction
• "Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation
• CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency