Self-Supervised Video Representation Learning by Video Incoherence Detection

Abstract

This paper introduces a novel self-supervised method that leverages incoherence detection for video representation learning. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between incoherent clips from the same raw video. We evaluate our proposed method through extensive experiments on action recognition and video retrieval utilizing various backbone networks. Experiments show that our proposed method achieves state-of-the-art performance across different backbone networks and different datasets compared with previous coherence-based methods.

arXiv comment11 pages, 7 figures

References in corpus (7)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Learning of Video Representations using LSTMs
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning