Learning Dense Correspondence via 3D-guided Cycle Consistency

Abstract

Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.

To appear in CVPR 2016 (oral presentation)

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
• Learning to See by Moving
• Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
• DeepStereo: Learning to Predict New Views from the World's Imagery
• Unsupervised Object Discovery and Localization in the Wild: Part-based Matching with Bottom-up Region Proposals
• Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
• Learning Deep Object Detectors from 3D Models
• Multi-Image Matching via Fast Alternating Minimization
• Virtual View Networks for Object Reconstruction

Cited by in corpus (14)

• Space-Time Correspondence as a Contrastive Random Walk
• Recurrent Transformer Networks for Semantic Correspondence
• Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
• Taming Visually Guided Sound Generation
• Learning Dense Facial Correspondences in Unconstrained Images
• FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence
• Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
• Surveillance Video Parsing with Single Frame Supervision
• PixelNN: Example-based Image Synthesis
• Multi-Image Semantic Matching by Mining Consistent Features
• ET-GAN: Cross-Language Emotion Transfer Based on Cycle-Consistent Generative Adversarial Networks
• DCTM: Discrete-Continuous Transformation Matching for Semantic Flow
• Semantic Attribute Matching Networks
• A gamified simulator and physical platform for self-driving algorithm training and validation