Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Abstract

We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self-supervised approaches have used spatial context [9, 34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30]. We propose to combine spatial and temporal context in one self-supervised framework without any heavy preprocessing. We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames. So the network is trained to correctly identify the position of a patch within a video frame as well as the position of a patch over time. We also propose a novel permutation strategy that outperforms random permutations while significantly reducing computational and memory constraints. We use our trained network for transfer learning tasks such as video activity recognition and demonstrate the strength of our approach on two benchmark video action recognition datasets without using a single frame from these datasets for unsupervised pretraining of our proposed video jigsaw network.

References in corpus (11)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• The Kinetics Human Action Video Dataset
• Generating Videos with Scene Dynamics
• Unsupervised Visual Representation Learning by Context Prediction
• Data-dependent Initializations of Convolutional Neural Networks
• Learning to See by Moving
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Learning Representations for Automatic Colorization
• Transitive Invariance for Self-supervised Visual Representation Learning
• Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery

Cited by in corpus (4)

• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Contrastive Learning with Adversarial Examples
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition