Learning models for visual 3D localization with implicit mapping

Abstract

We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels. The goal is to learn an implicit representation of the environment at a higher, more abstract level. We propose to use a generative approach based on Generative Query Networks (GQNs, Eslami et al. 2018), asking the following questions: 1) Can GQN capture more complex scenes than those it was originally demonstrated on? 2) Can GQN be used for localization in those scenes? To study this approach we consider procedurally generated Minecraft worlds, for which we can generate images of complex 3D scenes along with camera pose coordinates. We first show that GQNs, enhanced with a novel attention mechanism can capture the structure of 3D scenes in Minecraft, as evidenced by their samples. We then apply the models to the localization problem, comparing the results to a discriminative baseline, and comparing the ways each approach captures the task uncertainty.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Matching Networks for One Shot Learning
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Emergence of grid-like representations by training recurrent neural networks to perform spatial localization
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Unsupervised Predictive Memory in a Goal-Directed Agent
• BA-Net: Dense Bundle Adjustment Network
• Neural SLAM: Learning to Explore with External Memory
• Neural Map: Structured Memory for Deep Reinforcement Learning
• Generic 3D Representation via Pose Estimation and Matching
• Unifying Map and Landmark Based Representations for Visual Navigation
• Semi-parametric Topological Memory for Navigation
• Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
• VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization

Cited by in corpus (5)

• Attentive Neural Processes
• A Generative Map for Image-based Camera Localization
• Robustifying Sequential Neural Processes
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Incremental Scene Synthesis