Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency

Abstract

Recent works have advanced the performance of self-supervised representation learning by a large margin. The core among these methods is intra-image invariance learning. Two different transformations of one image instance are considered as a positive sample pair, where various tasks are designed to learn invariant representations by comparing the pair. Analogically, for video data, representations of frames from the same video are trained to be closer than frames from other videos, i.e. intra-video invariance. However, cross-video relation has barely been explored for visual representation learning. Unlike intra-video invariance, ground-truth labels of cross-video relation is usually unavailable without human labors. In this paper, we propose a novel contrastive learning method which explores the cross-video relation by using cycle-consistency for general image representation learning. This allows to collect positive sample pairs across different video instances, which we hypothesize will lead to higher-level semantics. We validate our method by transferring our image representation to multiple downstream tasks including visual object tracking, image classification, and action recognition. We show significant improvement over state-of-the-art contrastive learning methods. Project page is available at https://happywu.github.io/cycle_contrast_video.

References in corpus (24)

• A Simple Framework for Contrastive Learning of Visual Representations
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Adversarial Feature Learning
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Exploring Simple Siamese Representation Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Spatiotemporal Contrastive Video Representation Learning
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Video Representation Learning by Pace Prediction
• Self-supervised Learning for Video Correspondence Flow
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Cycle-Contrast for Self-Supervised Video Representation Learning
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning