FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

Abstract

We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

References in corpus (6)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT
• Learning to Compare Image Patches via Convolutional Neural Networks
• LIFT: Learned Invariant Feature Transform
• Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
• Learning Dense Correspondence via 3D-guided Cycle Consistency

Cited by in corpus (6)

• Recurrent Transformer Networks for Semantic Correspondence
• Matching neural paths: transfer from recognition to correspondence search
• DCTM: Discrete-Continuous Transformation Matching for Semantic Flow
• Semantic Attribute Matching Networks
• PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondence
• Reconstructing and grounding narrated instructional videos in 3D