Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework

Abstract

We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) framework, we can train spatio-temporal convolutional networks to learn video representations. There are many flexible options in our IIC framework and we conduct experiments by using several different configurations. Evaluations are conducted on video retrieval and video recognition tasks using the learned video representation. Our proposed IIC outperforms current state-of-the-art results by a large margin, such as 16.7% and 9.5% points improvements in top-1 accuracy on UCF101 and HMDB51 datasets for video retrieval, respectively. For video recognition, improvements can also be obtained on these two benchmark datasets. Code is available at https://github.com/BestJuly/Inter-intra-video-contrastive-learning.

Accepted by ACMMM 2020. Our project page is at https://bestjuly.github.io/Inter-intra-video-contrastive-learning/

References in corpus (10)

• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• In Defense of the Triplet Loss for Person Re-Identification
• The Kinetics Human Action Video Dataset
• Unsupervised Representation Learning by Predicting Image Rotations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Multiview Coding
• Spatiotemporal Residual Networks for Video Action Recognition
• Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
• Rethinking Motion Representation: Residual Frames with 3D ConvNets for Better Action Recognition

Cited by in corpus (16)

• Hard Negative Mixing for Contrastive Learning
• TCLR: Temporal Contrastive Learning for Video Representation
• Self-Supervised Learning for Videos: A Survey
• TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning
• Dual Contrastive Learning for Spatio-temporal Representation
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning
• Hierarchical Contrastive Motion Learning for Video Action Recognition
• Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval
• Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning
• Recent Advances and Challenges in Deep Audio-Visual Correlation Learning
• Motion-Focused Contrastive Learning of Video Representations
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Inter-intra Variant Dual Representations forSelf-supervised Video Recognition
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation