Self-supervised Co-training for Video Representation Learning

Abstract

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.

arXiv commentNeurIPS2020

References in corpus (17)

• A Simple Framework for Contrastive Learning of Visual Representations
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Exploring the Limits of Language Modeling
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Supervised Contrastive Learning
• Spatiotemporal Contrastive Video Representation Learning
• Learning visual groups from co-occurrences in space and time
• Self-supervised Learning for Video Correspondence Flow
• Self-Supervised Video Representation Learning With Odd-One-Out Networks

Cited by in corpus (52)

• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• TCLR: Temporal Contrastive Learning for Video Representation
• Self-Supervised Learning for Videos: A Survey
• A Comprehensive Study of Deep Video Action Recognition
• HyperNet: Self-Supervised Hyperspectral Spatial-Spectral Feature Understanding Network for Hyperspectral Change Detection
• Spatiotemporal Contrastive Video Representation Learning
• Video Representation Learning with Visual Tempo Consistency
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition
• Dual Contrastive Learning for Spatio-temporal Representation
• 3D Human Action Representation Learning via Cross-View Consistency Pursuit
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning
• A Co-training Approach for Noisy Time Series Learning
• Socially-Aware Self-Supervised Tri-Training for Recommendation
• Video 3D Sampling for Self-supervised Representation Learning
• Self-Supervised Graph Co-Training for Session-based Recommendation
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• Relation Modeling in Spatio-Temporal Action Localization
• Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Self-supervised Video-centralised Transformer for Video Face Clustering
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• Self-supervised Video Object Segmentation by Motion Grouping
• Motion-Focused Contrastive Learning of Video Representations
• Universal Facial Encoding of Codec Avatars from VR Headsets
• Center-wise Local Image Mixture For Contrastive Representation Learning
• SleepPriorCL: Contrastive Representation Learning with Prior Knowledge-based Positive Mining and Adaptive Temperature for Sleep Staging
• Learning Cross-modal Contrastive Features for Video Domain Adaptation
• Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Exploring Stronger Feature for Temporal Action Localization
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Long-Short Temporal Contrastive Learning of Video Transformers
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation
• A Stronger Baseline for Ego-Centric Action Detection
• Unsupervised Person Re-identification via Simultaneous Clustering and Consistency Learning
• Inter-intra Variant Dual Representations forSelf-supervised Video Recognition
• Video Contrastive Learning with Global Context
• Boosting Supervised Learning Performance with Co-training
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Unsupervised Visual Representation Learning by Tracking Patches in Video
• CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
• Long Short View Feature Decomposition via Contrastive Video Representation Learning
• V-SlowFast Network for Efficient Visual Sound Separation
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation
• Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations
• Learning Goals from Failure
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• PreViTS: Contrastive Pretraining with Video Tracking Supervision