GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Abstract

We propose GeoNet, a jointly unsupervised learning framework for monocular depth, optical flow and ego-motion estimation from videos. The three components are coupled by the nature of 3D scene geometry, jointly learned by our framework in an end-to-end manner. Specifically, geometric relationships are extracted over the predictions of individual modules and then combined as an image reconstruction loss, reasoning about static and dynamic scene parts separately. Furthermore, we propose an adaptive geometric consistency loss to increase robustness towards outliers and non-Lambertian regions, which resolves occlusions and texture ambiguities effectively. Experimentation on the KITTI driving dataset reveals that our scheme achieves state-of-the-art results in all of the three tasks, performing better than previously unsupervised methods and comparably with supervised ones.

arXiv commentAccepted to CVPR 2018; Code will be made available at https://github.com/yzcjtr/GeoNet

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Going Deeper with Convolutions
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Stacked Hourglass Networks for Human Pose Estimation
• SfM-Net: Learning of Structure and Motion from Video
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Unsupervised Learning of Depth and Ego-Motion from Video
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

Cited by in corpus (25)

• A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
• Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
• Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
• Two Stream Networks for Self-Supervised Ego-Motion Estimation
• Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding
• Unsupervised High-Resolution Depth Learning From Videos With Dual Networks
• Moving Indoor: Unsupervised Video Depth Learning in Challenging Environments
• Visuomotor Understanding for Representation Learning of Driving Scenes
• Self-Supervised Deep Visual Odometry with Online Adaptation
• WGANVO: Monocular Visual Odometry based on Generative Adversarial Networks
• S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation
• Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching
• Optical Flow in Dense Foggy Scenes using Semi-Supervised Learning
• Generalizing to the Open World: Deep Visual Odometry with Online Adaptation
• Learning the Depths of Moving People by Watching Frozen People
• Self-supervised Visual-LiDAR Odometry with Flip Consistency
• Unsupervised Object Detection with LiDAR Clues
• Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection
• S4-Net: Geometry-Consistent Semi-Supervised Semantic Segmentation
• Future Video Synthesis with Object Motion Prediction
• Single Image Depth Prediction with Wavelet Decomposition
• Sequential Learning of Visual Tracking and Mapping Using Unsupervised Deep Neural Networks
• Transformer Guided Geometry Model for Flow-Based Unsupervised Visual Odometry
• Guided Feature Selection for Deep Visual Odometry
• Functionally Modular and Interpretable Temporal Filtering for Robust Segmentation