Video Representation Learning by Dense Predictive Coding

Abstract

The objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition. We make three contributions: First, we introduce the Dense Predictive Coding (DPC) framework for self-supervised representation learning on videos. This learns a dense encoding of spatio-temporal blocks by recurrently predicting future representations; Second, we propose a curriculum training scheme to predict further into the future with progressively less temporal context. This encourages the model to only encode slowly varying spatial-temporal signals, therefore leading to semantic representations; Third, we evaluate the approach by first training the DPC model on the Kinetics-400 dataset with self-supervised learning, and then finetuning the representation on a downstream task, i.e. action recognition. With single stream (RGB only), DPC pretrained representations achieve state-of-the-art self-supervised performance on both UCF101(75.7% top1 acc) and HMDB51(35.7% top1 acc), outperforming all previous learning methods by a significant margin, and approaching the performance of a baseline pre-trained on ImageNet.

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Instance Normalization: The Missing Ingredient for Fast Stylization
• The Kinetics Human Action Video Dataset
• Unsupervised Learning of Video Representations using LSTMs
• SlowFast Networks for Video Recognition
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Learning visual groups from co-occurrences in space and time
• Self-supervised Learning for Video Correspondence Flow

Cited by in corpus (10)

• Learning Video Representations using Contrastive Bidirectional Transformer
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Are we done with ImageNet?
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Video Understanding as Machine Translation
• Self-supervised Video Object Segmentation
• VideoSSL: Semi-Supervised Learning for Video Classification
• Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization
• DTG-Net: Differentiated Teachers Guided Self-Supervised Video Action Recognition
• Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian