Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction

Abstract

Despite learning based methods showing promising results in single view depth estimation and visual odometry, most existing approaches treat the tasks in a supervised manner. Recent approaches to single view depth estimation explore the possibility of learning without full supervision via minimizing photometric error. In this paper, we explore the use of stereo sequences for learning depth and visual odometry. The use of stereo sequences enables the use of both spatial (between left-right pairs) and temporal (forward backward) photometric warp error, and constrains the scene depth and camera motion to be in a common, real-world scale. At test time our framework is able to estimate single view depth and two-view odometry from a monocular sequence. We also show how we can improve on a standard photometric warp loss by considering a warp of deep features. We show through extensive experiments that: (i) jointly training for single view depth and visual odometry improves depth prediction because of the additional constraint imposed on depths and achieves competitive results for visual odometry; (ii) deep feature-based warping loss improves upon simple photometric warp loss for both single view depth estimation and visual odometry. Our method outperforms existing learning based methods on the KITTI driving dataset in both tasks. The source code is available at https://github.com/Huangying-Zhan/Depth-VO-Feat

arXiv comment8 pages, 6 figures, CVPR 2018

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Learning Deconvolution Network for Semantic Segmentation
• SfM-Net: Learning of Structure and Motion from Video
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Universal Correspondence Network
• Unsupervised Learning of Depth and Ego-Motion from Video
• Learning to See by Moving
• LIFT: Learned Invariant Feature Transform
• Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Cited by in corpus (20)

• Monocular Depth Estimation with Self-supervised Instance Adaptation
• Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss
• Flexible Supervised Autonomy for Exploration in Subterranean Environments
• Joint Unsupervised Learning of Optical Flow and Depth by Watching Stereo Videos
• Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
• Two Stream Networks for Self-Supervised Ego-Motion Estimation
• Unsupervised High-Resolution Depth Learning From Videos With Dual Networks
• Active Safety Envelopes using Light Curtains with Probabilistic Guarantees
• Self-Supervised Deep Visual Odometry with Online Adaptation
• Structure-Attentioned Memory Network for Monocular Depth Estimation
• UnDEMoN 2.0: Improved Depth and Ego Motion Estimation through Deep Image Sampling
• Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video
• Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
• Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion
• Unsupervised Learning of Camera Pose with Compositional Re-estimation
• Generalizing to the Open World: Deep Visual Odometry with Online Adaptation
• Training Deep SLAM on Single Frames
• SIGNet: Semantic Instance Aided Unsupervised 3D Geometry Perception
• Self-supervised Visual-LiDAR Odometry with Flip Consistency
• SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation