A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence

Abstract

Deep learning based localization and mapping has recently attracted significant attention. Instead of creating hand-designed algorithms through exploitation of physical models or geometric theories, deep learning based solutions provide an alternative to solve the problem in a data-driven way. Benefiting from ever-increasing volumes of data and computational power, these methods are fast evolving into a new area that offers accurate and robust systems to track motion and estimate scenes and their structure for real-world applications. In this work, we provide a comprehensive survey, and propose a new taxonomy for localization and mapping using deep learning. We also discuss the limitations of current models, and indicate possible future directions. A wide range of topics are covered, from learning odometry estimation, mapping, to global localization and simultaneous localization and mapping (SLAM). We revisit the problem of perceiving self-motion and scene understanding with on-board sensors, and show how to solve it by integrating these modules into a prospective spatial machine intelligence system (SMIS). It is our hope that this work can connect emerging works from robotics, computer vision and machine learning communities, and serve as a guide for future researchers to apply deep learning to tackle localization and mapping problems.

arXiv comment26 pages, 10 figures. Project website: https://github.com/changhao-chen/deep-learning-localization-mapping

References in corpus (37)

• VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
• Language Models are Few-Shot Learners
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• On-Manifold Preintegration for Real-Time Visual-Inertial Odometry
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Learning to Navigate in Complex Environments
• R2D2: Repeatable and Reliable Detector and Descriptor
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Universal Correspondence Network
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• Convolutional Neural Network-based Place Recognition
• LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation
• Hierarchical Scene Coordinate Classification and Regression for Visual Localization
• On the Performance of ConvNet Features for Place Recognition
• BA-Net: Dense Bundle Adjustment Network
• LocNet: Global localization in 3D point clouds for mobile vehicles
• UnsuperPoint: End-to-end Unsupervised Interest Point Detector and Descriptor
• Learning to Localize Using a LiDAR Intensity Map
• Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters
• Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth
• FutureMapping: The Computational Structure of Spatial AI Systems
• D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features
• ASLFeat: Learning Local Features of Accurate Shape and Localization
• Geometry-Aware Neural Rendering
• To Learn or Not to Learn: Visual Localization from Essential Matrices
• D2D: Keypoint Extraction with Describe to Detect Approach
• DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints
• KFNet: Learning Temporal Camera Relocalization using Kalman Filtering
• Neural Multisensory Scene Inference
• Full-Frame Scene Coordinate Regression for Image-Based Localization
• AtLoc: Attention Guided Camera Localization

Cited by in corpus (19)

• A review of laser scanning for geological and geotechnical applications in underground mining
• Semantic localization on BIM-generated maps using a 3D LiDAR sensor
• RaLL: End-to-end Radar Localization on Lidar Map Using Differentiable Measurement Model
• Wide-Baseline Relative Camera Pose Estimation with Directional Learning
• A Comprehensive Survey of Machine Learning Based Localization with Wireless Signals
• TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation
• Deep Online Correction for Monocular Visual Odometry
• An Improved Multi-State Constraint Kalman Filter for Visual-Inertial Odometry
• MDN-VO: Estimating Visual Odometry with Confidence
• Multi-Camera Sensor Fusion for Visual Odometry using Deep Uncertainty Estimation
• CNN-based local features for navigation near an asteroid
• A Versatile Keyframe-Based Structureless Filter for Visual Inertial Odometry
• Improving the generalization of network based relative pose regression: dimension reduction as a regularizer
• Point wise or Feature wise? Benchmark Comparison of Public Available LiDAR Odometry Algorithms in Urban Canyons
• Information-Theoretic Odometry Learning
• Exploring Self-Attention for Visual Odometry
• The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation
• Salient Bundle Adjustment for Visual SLAM
• ATDN vSLAM: An all-through Deep Learning-Based Solution for Visual Simultaneous Localization and Mapping