Robustness Meets Deep Learning: An End-to-End Hybrid Pipeline for Unsupervised Learning of Egomotion

Abstract

In this work, we propose a method that combines unsupervised deep learning predictions for optical flow and monocular disparity with a model based optimization procedure for instantaneous camera pose. Given the flow and disparity predictions from the network, we apply a RANSAC outlier rejection scheme to find an inlier set of flows and disparities, which we use to solve for the relative camera pose in a least squares fashion. We show that this pipeline is fully differentiable, allowing us to combine the pose with the network outputs as an additional unsupervised training loss to further refine the predicted flows and disparities. This method not only allows us to directly regress relative pose from the network outputs, but also automatically segments away pixels that do not fit the rigid scene assumptions that many unsupervised structure from motion methods apply, such as on independently moving objects. We evaluate our method on the KITTI dataset, and demonstrate state of the art results, even in the presence of challenging independently moving objects.

arXiv comment10 pages, 5 figures, 7 tables

References in corpus (7)

• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• FlowNet: Learning Optical Flow with Convolutional Networks
• SfM-Net: Learning of Structure and Motion from Video
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Self-Supervised Learning for Stereo Matching with Self-Improving Ability
• Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding
• Recurrent Neural Network for Learning DenseDepth and Ego-Motion from Video

Cited by in corpus (6)

• DeepLO: Geometry-Aware Deep LiDAR Odometry
• Self-Supervised Deep Pose Corrections for Robust Visual Odometry
• Two Stream Networks for Self-Supervised Ego-Motion Estimation
• Self-Supervised 3D Keypoint Learning for Ego-motion Estimation
• Self-supervised Visual-LiDAR Odometry with Flip Consistency
• Sequential Learning of Visual Tracking and Mapping Using Unsupervised Deep Neural Networks