R2D2: Repeatable and Reliable Detector and Descriptor

Abstract

Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.

References in corpus (6)

• Learning Deep Embeddings with Histogram Loss
• Universal Correspondence Network
• PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors
• Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters
• BinGAN: Learning Compact Binary Descriptors with a Regularized GAN
• From handcrafted to deep local features

Cited by in corpus (44)

• Image Matching across Wide Baselines: From Paper to Practice
• DISK: Learning local features with policy gradient
• Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis
• A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
• Dual-Resolution Correspondence Networks
• LoFTR: Detector-Free Local Feature Matching with Transformers
• TopicFM: Robust and Interpretable Topic-Assisted Feature Matching
• Line as a Visual Sentence: Context-aware Line Descriptor for Visual Localization
• Unifying Deep Local and Global Features for Image Search
• UR2KiD: Unifying Retrieval, Keypoint Detection, and Keypoint Description without Local Correspondence Supervision
• D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features
• Probabilistic Spatial Distribution Prior Based Attentional Keypoints Matching Network
• Robust Image Retrieval-based Visual Localization using Kapture
• Knowledge Distillation for Feature Extraction in Underwater VSLAM
• Image Stylization for Robust Features
• TopicFM+: Boosting Accuracy and Efficiency of Topic-Assisted Feature Matching
• Robust Partial-to-Partial Point Cloud Registration in a Full Range
• HyNet: Learning Local Descriptor with Hybrid Similarity Measure and Triplet Loss
• Learnable Motion Coherence for Correspondence Pruning
• Improving the HardNet Descriptor
• Evaluation of Point Pattern Features for Anomaly Detection of Defect within Random Finite Set Framework
• COTR: Correspondence Transformer for Matching Across Images
• S2DNet: Learning Accurate Correspondences for Sparse-to-Dense Feature Matching
• PREDATOR: Registration of 3D Point Clouds with Low Overlap
• Self-Supervised 3D Keypoint Learning for Ego-motion Estimation
• Visual Localization Using Semantic Segmentation and Depth Prediction
• AdaLAM: Revisiting Handcrafted Outlier Detection
• StickyPillars: Robust and Efficient Feature Matching on Point Clouds using Graph Neural Networks
• Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions
• D2D: Keypoint Extraction with Describe to Detect Approach
• Learning Feature Descriptors using Camera Pose Supervision
• RANSAC-Flow: generic two-stage image alignment
• SOLD2: Self-supervised Occlusion-aware Line Description and Detection
• SDGMNet: Statistic-based Dynamic Gradient Modulation for Local Descriptor Learning
• Realtime CNN-based Keypoint Detector with Sobel Filter and CNN-based Descriptor Trained with Keypoint Candidates
• Soft Expectation and Deep Maximization for Image Feature Detection
• Retrieval and Localization with Observation Constraints
• Controllable Data Augmentation Through Deep Relighting
• Discriminative and Semantic Feature Selection for Place Recognition towards Dynamic Environments
• Learning Local Features with Context Aggregation for Visual Localization
• ViewSynth: Learning Local Features from Depth using View Synthesis
• Leveraging Local and Global Descriptors in Parallel to Search Correspondences for Visual Localization
• Reconstructing and grounding narrated instructional videos in 3D
• Pose Refinement with Joint Optimization of Visual Points and Lines