Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions

Abstract

In this work we target the problem of estimating accurately localised correspondences between a pair of images. We adopt the recent Neighbourhood Consensus Networks that have demonstrated promising performance for difficult correspondence problems and propose modifications to overcome their main limitations: large memory consumption, large inference time and poorly localised correspondences. Our proposed modifications can reduce the memory footprint and execution time more than $10\times$, with equivalent results. This is achieved by sparsifying the correlation tensor containing tentative matches, and its subsequent processing with a 4D CNN using submanifold sparse convolutions. Localisation accuracy is significantly improved by processing the input images in higher resolution, which is possible due to the reduced memory footprint, and by a novel two-stage correspondence relocalisation module. The proposed Sparse-NCNet method obtains state-of-the-art results on the HPatches Sequences and InLoc visual localisation benchmarks, and competitive results in the Aachen Day-Night benchmark.

References in corpus (6)

• R2D2: Repeatable and Reliable Detector and Descriptor
• TILDE: A Temporally Invariant Learned DEtector
• Spatially-sparse convolutional neural networks
• PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors
• Learning Two-View Correspondences and Geometry Using Order-Aware Network
• Sparse 3D convolutional neural networks

Cited by in corpus (4)

• TopicFM: Robust and Interpretable Topic-Assisted Feature Matching
• HyNet: Learning Local Descriptor with Hybrid Similarity Measure and Triplet Loss
• Learnable Motion Coherence for Correspondence Pruning
• Improving the generalization of network based relative pose regression: dimension reduction as a regularizer