ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency

Abstract

We study self-supervised video representation learning, which is a challenging task due to 1) lack of labels for explicit supervision; 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other, but they need a careful treatment of negative pairs by either relying on large batch sizes, memory banks, extra modalities or customized mining strategies, which inevitably includes noisy data. In this paper, we observe that the consistency between positive samples is the key to learn robust video representation. Specifically, we propose two tasks to learn the appearance and speed consistency, respectively. The appearance consistency task aims to maximize the similarity between two clips of the same video with different playback speeds. The speed consistency task aims to maximize the similarity between two clips with the same playback speed but different appearance information. We show that optimizing the two tasks jointly consistently improves the performance on downstream tasks, e.g., action recognition and video retrieval. Remarkably, for action recognition on the UCF-101 dataset, we achieve 90.8\% accuracy without using any extra modalities or negative pairs for unsupervised pretraining, which outperforms the ImageNet supervised pretrained model. Codes and models will be available.

Accepted to ICCV2021

References in corpus (14)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Exploring Simple Siamese Representation Learning
• Self-supervised Co-training for Video Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Spatiotemporal Contrastive Video Representation Learning
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Self-supervised Video Representation Learning by Pace Prediction
• Are all negatives created equal in contrastive instance discrimination?
• Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
• RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning