Self-supervised Visual-LiDAR Odometry with Flip Consistency

Abstract

Most learning-based methods estimate ego-motion by utilizing visual sensors, which suffer from dramatic lighting variations and textureless scenarios. In this paper, we incorporate sparse but accurate depth measurements obtained from lidars to overcome the limitation of visual methods. To this end, we design a self-supervised visual-lidar odometry (Self-VLO) framework. It takes both monocular images and sparse depth maps projected from 3D lidar points as input, and produces pose and depth estimations in an end-to-end learning manner, without using any ground truth labels. To effectively fuse two modalities, we design a two-pathway encoder to extract features from visual and depth images and fuse the encoded features with those in decoders at multiple scales by our fusion module. We also adopt a siamese architecture and design an adaptively weighted flip consistency loss to facilitate the self-supervised learning of our VLO. Experiments on the KITTI odometry benchmark show that the proposed approach outperforms all self-supervised visual or lidar odometries. It also performs better than fully supervised VOs, demonstrating the power of fusion.

9 pages, 5 figures, 2 tables, WACV 2021

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• Unsupervised Learning of Depth and Ego-Motion from Video
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction
• D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
• Learning Guided Convolutional Network for Depth Completion
• Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry
• Robustness Meets Deep Learning: An End-to-End Hybrid Pipeline for Unsupervised Learning of Egomotion
• Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
• LiDAR Data Enrichment Using Deep Learning Based on High-Resolution Image: An Approach to Achieve High-Performance LiDAR SLAM Using Low-cost LiDAR