Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry

Abstract

In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at: https://github.com/guangmingw/DOPlearning.

arXiv commentPublished in: IEEE Transactions on Intelligent Transportation Systems. DOI: 10.1109/TITS.2020.3010418

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks
• Unsupervised Learning of Monocular Depth and Ego-Motion Using Multiple Masks

Cited by in corpus (6)

• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• Unsupervised Learning of 3D Scene Flow from Monocular Camera
• PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization
• NccFlow: Unsupervised Learning of Optical Flow With Non-occlusion from Geometry
• Unsupervised Monocular Depth Perception: Focusing on Moving Objects