VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry

Abstract

Semantic understanding and localization are fundamental enablers of robot autonomy that have for the most part been tackled as disjoint problems. While deep learning has enabled recent breakthroughs across a wide spectrum of scene understanding tasks, its applicability to state estimation tasks has been limited due to the direct formulation that renders it incapable of encoding scene-specific constrains. In this work, we propose the VLocNet++ architecture that employs a multitask learning approach to exploit the inter-task relationship between learning semantics, regressing 6-DoF global pose and odometry, for the mutual benefit of each of these tasks. Our network overcomes the aforementioned limitation by simultaneously embedding geometric and semantic knowledge of the world into the pose regression network. We propose a novel adaptive weighted fusion layer to aggregate motion-specific temporal information and to fuse semantic features into the localization stream based on region activations. Furthermore, we propose a self-supervised warping technique that uses the relative motion to warp intermediate network representations in the segmentation stream for learning consistent semantics. Finally, we introduce a first-of-a-kind urban outdoor localization dataset with pixel-level semantic labels and multiple loops for training deep networks. Extensive experiments on the challenging Microsoft 7-Scenes benchmark and our DeepLoc dataset demonstrate that our approach exceeds the state-of-the-art outperforming local feature-based methods while simultaneously performing multiple tasks and exhibiting substantial robustness in challenging scenarios.

arXiv commentDemo and dataset available at http://deeploc.cs.uni-freiburg.de

References in corpus (2)

• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Multi-task CNN Model for Attribute Prediction

Cited by in corpus (13)

• A Survey of Deep Learning Techniques for Autonomous Driving
• Self-Supervised Model Adaptation for Multimodal Semantic Segmentation
• Approaches, Challenges, and Applications for Deep Visual Odometry: Toward to Complicated and Emerging Areas
• CMRNet: Camera to LiDAR-Map Registration
• Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascade
• PADLoC: LiDAR-Based Deep Loop Closure Detection and Registration Using Panoptic Attention
• ViPR: Visual-Odometry-aided Pose Regression for 6DoF Camera Localization
• Pit30M: A Benchmark for Global Localization in the Age of Self-Driving Cars
• Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments
• CoPR: Towards Accurate Visual Localization With Continuous Place-descriptor Regression
• HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR
• Initialisation of Autonomous Aircraft Visual Inspection Systems via CNN-Based Camera Pose Estimation
• CNN-Based Camera Pose Estimation and Localisation of Scan Images for Aircraft Visual Inspection