Video 3D Sampling for Self-supervised Representation Learning

Abstract

Most of the existing video self-supervised methods mainly leverage temporal signals of videos, ignoring that the semantics of moving objects and environmental information are all critical for video-related tasks. In this paper, we propose a novel self-supervised method for video representation learning, referred to as Video 3D Sampling (V3S). In order to sufficiently utilize the information (spatial and temporal) provided in videos, we pre-process a video from three dimensions (width, height, time). As a result, we can leverage the spatial information (the size of objects), temporal information (the direction and magnitude of motions) as our learning target. In our implementation, we combine the sampling of the three dimensions and propose the scale and projection transformations in space and time respectively. The experimental results show that, when applied to action recognition, video retrieval and action similarity labeling, our approach improves the state-of-the-arts with significant margins.

arXiv comment9 pages, 5 figures, 6 tables

References in corpus (5)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction

Cited by in corpus (3)

• Dense Semantic Contrast for Self-Supervised Visual Representation Learning
• Mask is All You Need: Rethinking Mask R-CNN for Dense and Arbitrary-Shaped Scene Text Detection
• PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition