A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
arXiv:2104.14558
Abstract
We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code is made available at https://github.com/facebookresearch/SlowFast
arXiv commentCVPR 2021
References in corpus (29)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- A Simple Framework for Contrastive Learning of Visual Representations
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Learning deep representations by mutual information estimation and maximization
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Unsupervised Learning of Video Representations using LSTMs
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Large Batch Training of Convolutional Networks
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
- A Short Note about Kinetics-600
- Self-supervised Co-training for Video Representation Learning
- Learning Video Representations using Contrastive Bidirectional Transformer
- Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- The AVA-Kinetics Localized Human Actions Video Dataset
- Spatiotemporal Contrastive Video Representation Learning
- Learning visual groups from co-occurrences in space and time
- Video Representation Learning with Visual Tempo Consistency
- Watching the World Go By: Representation Learning from Unlabeled Videos
- Learning Spatiotemporal Features via Video and Text Pair Discrimination
- BYOL works even without batch statistics
- Self-Supervised Video Representation Learning With Odd-One-Out Networks