Video Contrastive Learning with Global Context

Abstract

Contrastive learning has revolutionized self-supervised image representation learning field, and recently been adapted to video domain. One of the greatest advantages of contrastive learning is that it allows us to flexibly define powerful loss objectives as long as we can find a reasonable way to formulate positive and negative samples to contrast. However, existing approaches rely heavily on the short-range spatiotemporal salience to form clip-level contrastive signals, thus limit themselves from using global context. In this paper, we propose a new video-level contrastive learning method based on segments to formulate positive pairs. Our formulation is able to capture global context in a video, thus robust to temporal content change. We also incorporate a temporal order regularization term to enforce the inherent sequential structure of videos. Extensive experiments show that our video-level contrastive learning framework (VCLR) is able to outperform previous state-of-the-arts on five video datasets for downstream action classification, action localization and video retrieval. Code is available at https://github.com/amazon-research/video-contrastive-learning.

arXiv commentCode is publicly available at: https://github.com/amazon-research/video-contrastive-learning

References in corpus (22)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Kinetics Human Action Video Dataset
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Learning of Video Representations using LSTMs
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Self-supervised Co-training for Video Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• A Comprehensive Study of Deep Video Action Recognition
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• Video Representation Learning with Visual Tempo Consistency
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
• RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning
• Unsupervised Visual Representation Learning by Tracking Patches in Video