Video Representation Learning with Visual Tempo Consistency

Abstract

Visual tempo, which describes how fast an action goes, has shown its potential in supervised action recognition. In this work, we demonstrate that visual tempo can also serve as a self-supervision signal for video representation learning. We propose to maximize the mutual information between representations of slow and fast videos via hierarchical contrastive learning (VTHCL). Specifically, by sampling the same instance at slow and fast frame rates respectively, we can obtain slow and fast video frames which share the same semantics but contain different visual tempos. Video representations learned from VTHCL achieve the competitive performances under the self-supervision evaluation protocol for action recognition on UCF-101 (82.1\%) and HMDB-51 (49.2\%). Moreover, comprehensive experiments suggest that the learned representations are generalized well to other downstream tasks including action detection on AVA and action anticipation on Epic-Kitchen. Finally, we propose Instance Correspondence Map (ICM) to visualize the shared semantics captured by contrastive learning.

arXiv commentTechnical report. Models are available at https://github.com/decisionforce/VTHCL

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• How transferable are features in deep neural networks?
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• On the importance of single directions for generalization
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction

Cited by in corpus (17)

• TCLR: Temporal Contrastive Learning for Video Representation
• A Comprehensive Study of Deep Video Action Recognition
• Motion-driven Visual Tempo Learning for Video-based Action Recognition
• Spatiotemporal Contrastive Video Representation Learning
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Learning Video Representations from Textual Web Supervision
• Hierarchical Contrastive Motion Learning for Video Action Recognition
• Instance Localization for Self-supervised Detection Pretraining
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• Temporal Query Networks for Fine-grained Video Understanding
• Unsupervised Visual Representation Learning by Tracking Patches in Video
• Video Contrastive Learning with Global Context
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• Boosting Video Representation Learning with Multi-Faceted Integration
• On Compositions of Transformations in Contrastive Self-Supervised Learning