Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
arXiv:2008.13426
Abstract
This paper proposes a novel pretext task to address the self-supervised video representation learning problem. Specifically, given an unlabeled video clip, we compute a series of spatio-temporal statistical summaries, such as the spatial location and dominant direction of the largest motion, the spatial location and dominant color of the largest color diversity along the temporal axis, etc. Then a neural network is built and trained to yield the statistical summaries given the video frames as inputs. In order to alleviate the learning difficulty, we employ several spatial partitioning patterns to encode rough spatial locations instead of exact spatial Cartesian coordinates. Our approach is inspired by the observation that human visual system is sensitive to rapidly changing contents in the visual field, and only needs impressions about rough spatial locations to understand the visual contents. To validate the effectiveness of the proposed approach, we conduct extensive experiments with four 3D backbone networks, i.e., C3D, 3D-ResNet, R(2+1)D and S3D-G. The results show that our approach outperforms the existing approaches across these backbone networks on four downstream video analysis tasks including action recognition, video retrieval, dynamic scene recognition, and action similarity labeling. The source code is publicly available at: https://github.com/laura-wang/video_repres_sts.
arXiv commentAccepted by TPAMI. An extension of our previous work at arXiv:1904.03597
References in corpus (15)
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Contrastive Multiview Coding
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
- What makes ImageNet good for transfer learning?
- Use What You Have: Video Retrieval Using Representations From Collaborative Experts
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- On The Power of Curriculum Learning in Training Deep Networks
- Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning