Unsupervised Visual Representation Learning by Tracking Patches in Video
arXiv:2105.02545
Abstract
Inspired by the fact that human eyes continue to develop tracking ability in early and middle childhood, we propose to use tracking as a proxy task for a computer vision system to learn the visual representations. Modelled on the Catch game played by the children, we design a Catch-the-Patch (CtP) game for a 3D-CNN model to learn visual representations that would help with video-related tasks. In the proposed pretraining framework, we cut an image patch from a given video and let it scale and move according to a pre-set trajectory. The proxy task is to estimate the position and size of the image patch in a sequence of video frames, given only the target bounding box in the first frame. We discover that using multiple image patches simultaneously brings clear benefits. We further increase the difficulty of the game by randomly making patches invisible. Extensive experiments on mainstream benchmarks demonstrate the superior performance of CtP against other video pretraining methods. In addition, CtP-pretrained features are less sensitive to domain gaps than those trained by a supervised action recognition task. When both trained on Kinetics-400, we are pleasantly surprised to find that CtP-pretrained representation achieves much higher action classification accuracy than its fully supervised counterpart on Something-Something dataset. Code is available online: github.com/microsoft/CtP.
arXiv commentTo appear in CVPR'21. Code available at github.com/microsoft/CtP
References in corpus (16)
- A Simple Framework for Contrastive Learning of Visual Representations
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Improved Regularization of Convolutional Neural Networks with Cutout
- GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
- Virtual Worlds as Proxy for Multi-Object Tracking Analysis
- Self-supervised Co-training for Video Representation Learning
- Learning Video Representations using Contrastive Bidirectional Transformer
- Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
- Memory-augmented Dense Predictive Coding for Video Representation Learning
- Video Representation Learning with Visual Tempo Consistency
- Self-supervised Video Representation Learning by Pace Prediction
- Self-supervised Learning for Video Correspondence Flow
- Joint-task Self-supervised Learning for Temporal Correspondence
- Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
- Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning