Learning Correspondence from the Cycle-Consistency of Time

Abstract

We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch. At training time, our model learns a feature map representation to be useful for performing cycle-consistent tracking. At test time, we use the acquired representation to find nearest neighbors across space and time. We demonstrate the generalizability of the representation -- without finetuning -- across a range of visual correspondence tasks, including video object segmentation, keypoint tracking, and optical flow. Our approach outperforms previous self-supervised methods and performs competitively with strongly supervised methods.

arXiv commentCVPR 2019 Oral. Project page: http://ajabri.github.io/timecycle

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FlowNet: Learning Optical Flow with Convolutional Networks
• Deep multi-scale video prediction beyond mean square error
• SfM-Net: Learning of Structure and Motion from Video
• Efficient Video Object Segmentation via Network Modulation
• Holistic, Instance-Level Human Parsing