VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change

Abstract

Visual Place Recognition (VPR) is the process of recognising a previously visited place using visual information, often under varying appearance conditions and viewpoint changes and with computational constraints. VPR is related to the concepts of localisation, loop closure, image retrieval and is a critical component of many autonomous navigation systems ranging from autonomous vehicles to drones and computer vision systems. While the concept of place recognition has been around for many years, VPR research has grown rapidly as a field over the past decade due to improving camera hardware and its potential for deep learning-based techniques, and has become a widely studied topic in both the computer vision and robotics communities. This growth however has led to fragmentation and a lack of standardisation in the field, especially concerning performance evaluation. Moreover, the notion of viewpoint and illumination invariance of VPR techniques has largely been assessed qualitatively and hence ambiguously in the past. In this paper, we address these gaps through a new comprehensive open-source framework for assessing the performance of VPR techniques, dubbed "VPR-Bench". VPR-Bench (Open-sourced at: https://github.com/MubarizZaffar/VPR-Bench) introduces two much-needed capabilities for VPR researchers: firstly, it contains a benchmark of 12 fully-integrated datasets and 10 VPR techniques, and secondly, it integrates a comprehensive variation-quantified dataset for quantifying viewpoint and illumination invariance. We apply and analyse popular evaluation metrics for VPR from both the computer vision and robotics communities, and discuss how these different metrics complement and/or replace each other, depending upon the underlying applications and system requirements.

Accepted version of our IJCV paper

References in corpus (8)

• Image Matching across Wide Baselines: From Paper to Practice
• Convolutional Neural Network-based Place Recognition
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Multi-Process Fusion: Visual Place Recognition Using Multiple Image Processing Methods
• A Hybrid Compact Neural Architecture for Visual Place Recognition
• Levelling the Playing Field: A Comprehensive Comparison of Visual Place Recognition Approaches under Changing Conditions
• Unifying Deep Local and Global Features for Image Search
• ConvSequential-SLAM: A Sequence-based, Training-less Visual Place Recognition Technique for Changing Environments

Cited by in corpus (8)

• The Revisiting Problem in Simultaneous Localization and Mapping: A Survey on Visual Loop Closure Detection
• Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments
• Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
• Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios
• EchoVPR: Echo State Networks for Visual Place Recognition
• Boosting Performance of a Baseline Visual Place Recognition Technique by Predicting the Maximally Complementary Technique
• Self-Supervised Place Recognition by Refining Temporal and Featural Pseudo Labels from Panoramic Data
• Sequence-Based Filtering for Visual Route-Based Navigation: Analysing the Benefits, Trade-offs and Design Choices