Dual Contrastive Learning for Spatio-temporal Representation

Abstract

Contrastive learning has shown promising potential in self-supervised spatio-temporal representation learning. Most works naively sample different clips to construct positive and negative pairs. However, we observe that this formulation inclines the model towards the background scene bias. The underlying reasons are twofold. First, the scene difference is usually more noticeable and easier to discriminate than the motion difference. Second, the clips sampled from the same video often share similar backgrounds but have distinct motions. Simply regarding them as positive pairs will draw the model to the static background rather than the motion pattern. To tackle this challenge, this paper presents a novel dual contrastive formulation. Concretely, we decouple the input RGB video sequence into two complementary modes, static scene and dynamic motion. Then, the original RGB features are pulled closer to the static features and the aligned dynamic features, respectively. In this way, the static scene and the dynamic motion are simultaneously encoded into the compact RGB representation. We further conduct the feature space decoupling via activation maps to distill static- and dynamic-related features. We term our method as \textbf{D}ual \textbf{C}ontrastive \textbf{L}earning for spatio-temporal \textbf{R}epresentation (DCLR). Extensive experiments demonstrate that DCLR learns effective spatio-temporal representations and obtains state-of-the-art or comparable performance on UCF-101, HMDB-51, and Diving-48 datasets.

arXiv commentACM MM 2022 camera ready

References in corpus (18)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Representation Learning by Predicting Image Rotations
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Space-Time Correspondence as a Contrastive Random Walk
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition
• What makes instance discrimination good for transfer learning?
• Cycle-Contrast for Self-Supervised Video Representation Learning
• Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning
• Bag of Instances Aggregation Boosts Self-supervised Distillation