Unsupervised Learning of Depth and Ego-Motion from Video

Abstract

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. We achieve this by simultaneously training depth and camera pose estimation networks using the task of view synthesis as the supervisory signal. The networks are thus coupled via the view synthesis objective during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performing comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performing favorably with established SLAM systems under comparable input settings.

Accepted to CVPR 2017. Project webpage: https://people.eecs.berkeley.edu/~tinghuiz/projects/SfMLearner/

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Deep Convolutional Inverse Graphics Network
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Unsupervised Learning of Visual Representations using Videos
• Learning to See by Moving
• Unsupervised Learning of 3D Structure from Images
• Semi-Supervised Deep Learning for Monocular Depth Map Prediction
• DeepStereo: Learning to Predict New Views from the World's Imagery
• 3D Shape Induction from 2D Views of Multiple Objects
• Learning Features by Watching Objects Move
• Unsupervised Learning of Spatiotemporally Coherent Metrics
• gvnn: Neural Network Library for Geometric Computer Vision

Cited by in corpus (95)

• SfM-Net: Learning of Structure and Motion from Video
• Learning a Multi-View Stereo Machine
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Self-supervised Learning of Motion Capture
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency
• Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image
• Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction
• Unifying Map and Landmark Based Representations for Visual Navigation
• MegaDepth: Learning Single-View Depth Prediction from Internet Photos
• Unsupervised Video Object Segmentation for Deep Reinforcement Learning
• RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints
• UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning
• Toward Geometric Deep SLAM
• Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• Learning Depth from Monocular Videos using Direct Methods
• D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
• FutureMapping: The Computational Structure of Spatial AI Systems
• Self-Improving Visual Odometry
• Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data
• Monocular Depth Estimation using Multi-Scale Continuous CRFs as Sequential Deep Networks
• DeepLO: Geometry-Aware Deep LiDAR Odometry
• Camera-based vehicle velocity estimation from monocular video
• HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation
• Transitive Invariance for Self-supervised Visual Representation Learning
• LEGO: Learning Edge with Geometry all at Once by Watching Videos
• Unsupervised Depth Estimation, 3D Face Rotation and Replacement
• 3D Photography using Context-aware Layered Depth Inpainting
• Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Semantic Flow for Fast and Accurate Scene Parsing
• Single View Stereo Matching
• RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis
• Geometry-aware Deep Network for Single-Image Novel View Synthesis
• Joint Unsupervised Learning of Optical Flow and Depth by Watching Stereo Videos
• DOVE: Learning Deformable 3D Objects by Watching Videos
• Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
• Unsupervised learning of object frames by dense equivariant image labelling
• Disentangling Factors of Variation by Mixing Them
• SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation
• X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation
• Moving Indoor: Unsupervised Video Depth Learning in Challenging Environments
• Unsupervised High-Resolution Depth Learning From Videos With Dual Networks
• End-to-end depth from motion with stabilized monocular videos
• Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations
• InverseRenderNet: Learning single image inverse rendering
• Monocular Depth Estimators: Vulnerabilities and Attacks
• Visuomotor Understanding for Representation Learning of Driving Scenes
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
• PRGFlow: Benchmarking SWAP-Aware Unified Deep Visual Inertial Odometry
• Semi-supervised Learning: Fusion of Self-supervised, Supervised Learning, and Multimodal Cues for Tactical Driver Behavior Detection
• Unsupervised 3D Shape Learning from Image Collections in the Wild
• Layer-structured 3D Scene Inference via View Synthesis
• Future Person Localization in First-Person Videos
• Unsupervised Video Depth Estimation Based on Ego-motion and Disparity Consensus
• Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video
• Structure-Attentioned Memory Network for Monocular Depth Estimation
• Future Segmentation Using 3D Structure
• UnDEMoN 2.0: Improved Depth and Ego Motion Estimation through Deep Image Sampling
• Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion
• S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation
• Learning Multiple Dense Prediction Tasks from Partially Annotated Data
• Semantic Photometric Bundle Adjustment on Natural Sequences
• Unsupervised Representation Learning by Predicting Random Distances
• Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching
• Learning by Inertia: Self-supervised Monocular Visual Odometry for Road Vehicles
• Unsupervised Metric Relocalization Using Transform Consistency Loss
• Single Image Depth Estimation Trained via Depth from Defocus Cues
• SIGNet: Semantic Instance Aided Unsupervised 3D Geometry Perception
• Generalizing to the Open World: Deep Visual Odometry with Online Adaptation
• Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
• Self-supervised Visual-LiDAR Odometry with Flip Consistency
• Why Convolutional Networks Learn Oriented Bandpass Filters: Theory and Empirical Support
• Adversarial View-Consistent Learning for Monocular Depth Estimation
• Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation
• End-to-end Generative Zero-shot Learning via Few-shot Learning
• Robot Perception enables Complex Navigation Behavior via Self-Supervised Learning
• SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation
• Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos
• Learning Sports Camera Selection from Internet Videos
• Does it work outside this benchmark? Introducing the Rigid Depth Constructor tool, depth validation dataset construction in rigid scenes for the masses
• Deep Weakly Supervised Positioning
• Multi-Hypothesis Visual-Inertial Flow
• Multi range Real-time depth inference from a monocular stabilized footage using a Fully Convolutional Neural Network
• Indoor GeoNet: Weakly Supervised Hybrid Learning for Depth and Pose Estimation
• LO-Net: Deep Real-time Lidar Odometry
• Unsupervised Learning of Depth and Deep Representation for Visual Odometry from Monocular Videos in a Metric Space
• Functionally Modular and Interpretable Temporal Filtering for Robust Segmentation
• Unsupervised monocular stereo matching
• Sequential Learning of Visual Tracking and Mapping Using Unsupervised Deep Neural Networks
• A Multi-Task Learning Approach for Meal Assessment
• Improving Self-Supervised Single View Depth Estimation by Masking Occlusion
• Learning to Align Images using Weak Geometric Supervision
• Monocular Depth Estimation with Directional Consistency by Deep Networks