Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis

Abstract

Visual Localization is one of the key enabling technologies for autonomous driving and augmented reality. High quality datasets with accurate 6 Degree-of-Freedom (DoF) reference poses are the foundation for benchmarking and improving existing methods. Traditionally, reference poses have been obtained via Structure-from-Motion (SfM). However, SfM itself relies on local features which are prone to fail when images were taken under different conditions, e.g., day/ night changes. At the same time, manually annotating feature correspondences is not scalable and potentially inaccurate. In this work, we propose a semi-automated approach to generate reference poses based on feature matching between renderings of a 3D model and real images via learned features. Given an initial pose estimate, our approach iteratively refines the pose based on feature matches against a rendering of the model from the current pose estimate. We significantly improve the nighttime reference poses of the popular Aachen Day-Night dataset, showing that state-of-the-art visual localization methods perform better (up to $47\%$) than predicted by the original reference poses. We extend the dataset with new nighttime test images, provide uncertainty estimates for our new reference poses, and introduce a new evaluation criterion. We will make our reference poses and our framework publicly available upon publication.

arXiv comment25 pages, 16 figures. Int J Comput Vis (2020)

References in corpus (24)

• ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
• Visual-Inertial Monocular SLAM with Map Reuse
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• R2D2: Repeatable and Reliable Detector and Descriptor
• LF-Net: Learning Local Features from Images
• Working hard to know your neighbor's margins: Local descriptor learning loss
• maplab: An Open Framework for Research in Visual-inertial Mapping and Localization
• BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
• Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascade
• D2-Net: A Trainable CNN for Joint Detection and Description of Local Features
• Learning Two-View Correspondences and Geometry Using Order-Aware Network
• Keyframe-Based Visual-Inertial Online SLAM with Relocalization
• Deep Learning Features at Scale for Visual Place Recognition
• UR2KiD: Unifying Retrieval, Keypoint Detection, and Keypoint Description without Local Correspondence Supervision
• Improved Visual Relocalization by Discovering Anchor Points
• Understanding the Limitations of CNN-based Absolute Camera Pose Regression
• Real-time Image-based 6-DOF Localization in Large-Scale Environments
• Large Scale Joint Semantic Re-Localisation and Scene Understanding via Globally Unique Instance Coordinate Regression
• Backtracking Regression Forests for Accurate Camera Relocalization
• On-the-Fly Adaptation of Regression Forests for Online Camera Relocalisation
• Image-based Localization using Hourglass Networks
• To Learn or Not to Learn: Visual Localization from Essential Matrices
• Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
• KFNet: Learning Temporal Camera Relocalization using Kalman Filtering

Cited by in corpus (13)

• ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning
• Local Feature Matching Using Deep Learning: A Survey
• LoFTR: Detector-Free Local Feature Matching with Transformers
• TopicFM: Robust and Interpretable Topic-Assisted Feature Matching
• Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark
• AirLoop: Lifelong Loop Closure Detection
• LENS: Localization enhanced by NeRF synthesis
• TopicFM+: Boosting Accuracy and Efficiency of Topic-Assisted Feature Matching
• Shared Coupling-bridge for Weakly Supervised Local Feature Learning
• CoPR: Towards Accurate Visual Localization With Continuous Place-descriptor Regression
• Dark Side Augmentation: Generating Diverse Night Examples for Metric Learning
• Robust Self-calibration of Focal Lengths from the Fundamental Matrix
• TP3M: Transformer-based Pseudo 3D Image Matching with Reference Image