Self-supervised Video Representation Learning by Context and Motion Decoupling

Abstract

A key challenge in self-supervised video representation learning is how to effectively capture motion information besides context bias. While most existing works implicitly achieve this with video-specific pretext tasks (e.g., predicting clip orders, time arrows, and paces), we develop a method that explicitly decouples motion supervision from context bias through a carefully designed pretext task. Specifically, we take the keyframes and motion vectors in compressed videos (e.g., in H.264 format) as the supervision sources for context and motion, respectively, which can be efficiently extracted at over 500 fps on the CPU. Then we design two pretext tasks that are jointly optimized: a context matching task where a pairwise contrastive loss is cast between video clip and keyframe features; and a motion prediction task where clip features, passed through an encoder-decoder network, are used to estimate motion features in a near future. These two tasks use a shared video backbone and separate MLP heads. Experiments show that our approach improves the quality of the learned video representation over previous works, where we obtain absolute gains of 16.0% and 11.1% in video retrieval recall on UCF101 and HMDB51, respectively. Moreover, we find the motion prediction to be a strong regularization for video networks, where using it as an auxiliary task improves the accuracy of action recognition with a margin of 7.4%~13.8%.

Accepted by CVPR2021

References in corpus (20)

• A Simple Framework for Contrastive Learning of Visual Representations
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Audiovisual SlowFast Networks for Video Recognition
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Spatiotemporal Contrastive Video Representation Learning
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Video Representation Learning with Visual Tempo Consistency
• Self-supervised Video Representation Learning by Pace Prediction
• Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
• Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion
• Self-supervised Temporal Discriminative Learning for Video Representation Learning
• Fast Object Detection in Compressed Video