Joint-task Self-supervised Learning for Temporal Correspondence
arXiv:1909.11895
Abstract
This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames. We exploit the synergy between both tasks through a shared inter-frame affinity matrix, which simultaneously models transitions between video frames at both the region- and pixel-levels. While region-level localization helps reduce ambiguities in fine-grained matching by narrowing down search regions; fine-grained matching provides bottom-up features to facilitate region-level localization. Our method outperforms the state-of-the-art self-supervised methods on a variety of visual correspondence tasks, including video-object and part-segmentation propagation, keypoint tracking, and object tracking. Our self-supervised method even surpasses the fully-supervised affinity feature representation obtained from a ResNet-18 pre-trained on the ImageNet.
NeurIPS 2019
Cited by in corpus (9)
- A Comprehensive Study of Deep Video Action Recognition
- Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
- MAST: A Memory-Augmented Self-supervised Tracker
- Self-supervised Video Object Segmentation
- Reference-Based Video Colorization with Spatiotemporal Correspondence
- Automatic Horizontal Fusion for GPU Kernels
- Learning to Track Instances without Video Annotations
- Self-supervised Object Tracking with Cycle-consistent Siamese Networks
- Self-supervised Regularization for Text Classification