Space-Time Correspondence as a Contrastive Random Walk

Abstract

This paper proposes a simple self-supervised approach for learning a representation for visual correspondence from raw video. We cast correspondence as prediction of links in a space-time graph constructed from video. In this graph, the nodes are patches sampled from each frame, and nodes adjacent in time can share a directed edge. We learn a representation in which pairwise similarity defines transition probability of a random walk, so that long-range correspondence is computed as a walk along the graph. We optimize the representation to place high probability along paths of similarity. Targets for learning are formed without supervision, by cycle-consistency: the objective is to maximize the likelihood of returning to the initial node when walking along a graph constructed from a palindrome of frames. Thus, a single path-level constraint implicitly supervises chains of intermediate comparisons. When used as a similarity metric without adaptation, the learned representation outperforms the self-supervised state-of-the-art on label propagation tasks involving objects, semantic parts, and pose. Moreover, we demonstrate that a technique we call edge dropout, as well as self-supervised adaptation at test-time, further improve transfer for object-centric correspondence.

arXiv commentNeurIPS 2020 camera ready version -- Code at github.com/ajabri/videowalk

References in corpus (22)

• Efficient Estimation of Word Representations in Vector Space
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Deep multi-scale video prediction beyond mean square error
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Multi-Object Representation Learning with Iterative Variational Inference
• Unsupervised Learning by Predicting Noise
• Gromov-Wasserstein Learning for Graph Matching and Node Embedding
• Learning visual groups from co-occurrences in space and time
• Self-supervised Learning for Video Correspondence Flow
• Joint-task Self-supervised Learning for Temporal Correspondence
• Efficient Video Object Segmentation via Network Modulation
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• Learning Dense Correspondence via 3D-guided Cycle Consistency
• Multigrid Predictive Filter Flow for Unsupervised Learning on Videos
• Online Object Representations with Contrastive Learning
• MAST: A Memory-Augmented Self-supervised Tracker

Cited by in corpus (10)

• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
• Deep Learning Technique for Human Parsing: A Survey and Outlook
• Dual Contrastive Learning for Spatio-temporal Representation
• HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images
• Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
• Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
• Leveraging Cycle-Consistent Anchor Points for Self-Supervised RGB-D Registration
• Self-Distilled Self-Supervised Representation Learning
• Combining Supervised and Un-supervised Learning for Automatic Citrus Segmentation
• SFTrack++: A Fast Learnable Spectral Segmentation Approach for Space-Time Consistent Tracking