DeMoN: Depth and Motion Network for Learning Monocular Stereo
arXiv:1612.02401 · doi:10.1109/CVPR.2017.596
Abstract
In this paper we formulate structure from motion as a learning problem. We train a convolutional network end-to-end to compute depth and camera motion from successive, unconstrained image pairs. The architecture is composed of multiple stacked encoder-decoder networks, the core part being an iterative network that is able to improve its own predictions. The network estimates not only depth and motion, but additionally surface normals, optical flow between the images and confidence of the matching. A crucial component of the approach is a training loss based on spatial relative differences. Compared to traditional two-frame structure from motion methods, results are more accurate and more robust. In contrast to the popular depth-from-single-image networks, DeMoN learns the concept of matching and, thus, better generalizes to structures not seen during training.
arXiv commentCamera ready version for CVPR 2017. Supplementary material included. Project page: http://lmb.informatik.uni-freiburg.de/people/ummenhof/depthmotionnet/
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Caffe: Convolutional Architecture for Fast Feature Embedding
- A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
- Computing the Stereo Matching Cost with a Convolutional Neural Network
Cited by in corpus (165)
- Artificial Intelligence in the Creative Industries: A Review
- AdaBins: Depth Estimation using Adaptive Bins
- The ApolloScape Open Dataset for Autonomous Driving and its Application
- A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
- DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration
- DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
- Unsupervised Learning of Depth and Ego-Motion from Video
- High Quality Monocular Depth Estimation via Transfer Learning
- DeepFactors: Real-Time Probabilistic Dense Monocular SLAM
- DeepV2D: Video to Depth with Differentiable Structure from Motion
- BA-Net: Dense Bundle Adjustment Network
- Computer vision in automated parking systems: Design, implementation and challenges
- GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
- A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
- Approaches, Challenges, and Applications for Deep Visual Odometry: Toward to Complicated and Emerging Areas
- Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction
- Densely Connected Pyramid Dehazing Network
- LS-VO: Learning Dense Optical Subspace for Robust Visual Odometry Estimation
- Two-shot Spatially-varying BRDF and Shape Estimation
- MegaDepth: Learning Single-View Depth Prediction from Internet Photos
- DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network
- Unsupervised Monocular Depth Learning in Dynamic Scenes
- On the Continuity of Rotation Representations in Neural Networks
- Learning Two-View Correspondences and Geometry Using Order-Aware Network
- UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning
- Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
- Wide-Baseline Relative Camera Pose Estimation with Directional Learning
- Occlusion-Aware Depth Estimation with Adaptive Normal Constraints
- A Survey on RGB-D Datasets
- Toward Geometric Deep SLAM
- Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth
- Joint Transmission Map Estimation and Dehazing using Deep Networks
- Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
- D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
- Just-in-Time Reconstruction: Inpainting Sparse Maps using Single View Depth Predictors as Priors
- SAFENet: Self-Supervised Monocular Depth Estimation with Semantic-Aware Feature Extraction
- DeepMVS: Learning Multi-view Stereopsis
- Motion-based Camera Localization System in Colonoscopy Videos
- DF-VO: What Should Be Learnt for Visual Odometry?
- Detecting Photoshopped Faces by Scripting Photoshop
- A Survey on Deep Learning Architectures for Image-based Depth Reconstruction
- Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios
- Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera
- LS-Net: Learning to Solve Nonlinear Least Squares for Monocular Stereo
- Understanding the Limitations of CNN-based Absolute Camera Pose Regression
- Photorealistic Image Reconstruction from Hybrid Intensity and Event based Sensor
- TrimBot2020: an outdoor robot for automatic gardening
- DurLAR: A High-fidelity 128-channel LiDAR Dataset with Panoramic Ambient and Reflectivity Imagery for Multi-modal Autonomous Driving Applications
- Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries
- Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss
- Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
- Lightweight Monocular Depth Estimation Model by Joint End-to-End Filter pruning
- Relative Camera Pose Estimation Using Convolutional Neural Networks
- MVS^2: Deep Unsupervised Multi-view Stereo with Multi-View Symmetry
- Learning models for visual 3D localization with implicit mapping
- Jointly Adversarial Network to Wavelength Compensation and Dehazing of Underwater Images
- Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
- Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry
- RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
- Pyramid Feature Attention Network for Monocular Depth Prediction
- Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
- Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
- RealMonoDepth: Self-Supervised Monocular Depth Estimation for General Scenes
- Visual Odometry Revisited: What Should Be Learnt?
- ENG: End-to-end Neural Geometry for Robust Depth and Pose Estimation using CNNs
- VisualEchoes: Spatial Image Representation Learning through Echolocation
- M^3VSNet: Unsupervised Multi-metric Multi-view Stereo Network
- ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning
- Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
- To Learn or Not to Learn: Visual Localization from Essential Matrices
- Local Supports Global: Deep Camera Relocalization with Sequence Enhancement
- milliEgo: Single-chip mmWave Radar Aided Egomotion Estimation via Deep Sensor Fusion
- On the Coupling of Depth and Egomotion Networks for Self-Supervised Structure from Motion
- Consistent Video Depth Estimation
- Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
- Recurrent Neural Network for Learning DenseDepth and Ego-Motion from Video
- Image Based Camera Localization: an Overview
- Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications
- Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference
- Robust Semi-Supervised Monocular Depth Estimation with Reprojected Distances
- COTR: Correspondence Transformer for Matching Across Images
- InverseRenderNet: Learning single image inverse rendering
- Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
- Self-supervised Learning for Single View Depth and Surface Normal Estimation
- DeLS-3D: Deep Localization and Segmentation with a 3D Semantic Map
- Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes
- CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
- Beyond Photometric Loss for Self-Supervised Ego-Motion Estimation
- MonSter: Awakening the Mono in Stereo
- TartanVO: A Generalizable Learning-based VO
- DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds
- Eigendecomposition-free Training of Deep Networks with Zero Eigenvalue-based Losses
- End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization
- Learning to Adapt Multi-View Stereo by Self-Supervision
- LIFE: Lighting Invariant Flow Estimation
- NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
- MobileDepth: Efficient Monocular Depth Prediction on Mobile Devices
- Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains
- Extreme Relative Pose Network under Hybrid Representations
- DeepSFM: Structure From Motion Via Deep Bundle Adjustment
- Online Mutual Adaptation of Deep Depth Prediction and Visual SLAM
- Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
- Deep Model-Based 6D Pose Refinement in RGB
- UnDEMoN 2.0: Improved Depth and Ego Motion Estimation through Deep Image Sampling
- MVS2D: Efficient Multi-view Stereo via Attention-Driven 2D Convolutions
- Generative View Synthesis: From Single-view Semantics to Novel-view Images
- Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video
- Scene Motion Decomposition for Learnable Visual Odometry
- Learning Depth With Very Sparse Supervision
- Normal Assisted Stereo Depth Estimation
- Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling
- S-BEV: Semantic Birds-Eye View Representation for Weather and Lighting Invariant 3-DoF Localization
- Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
- Robust Consistent Video Depth Estimation
- Binary TTC: A Temporal Geofence for Autonomous Navigation
- Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
- Self-Supervised Learning of Depth and Motion Under Photometric Inconsistency
- MonoIndoor: Towards Good Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments
- Taking a Deeper Look at the Inverse Compositional Algorithm
- Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images
- Learning structure-from-motion from motion
- DeepTAM: Deep Tracking and Mapping
- PoseGAN: A Pose-to-Image Translation Framework for Camera Localization
- Improving Monocular Depth Estimation by Leveraging Structural Awareness and Complementary Datasets
- Unsupervised Learning of Camera Pose with Compositional Re-estimation
- Learning the Depths of Moving People by Watching Frozen People
- Learning by Inertia: Self-supervised Monocular Visual Odometry for Road Vehicles
- Improving Depth Estimation using Location Information
- Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty
- Monocular Depth Prediction through Continuous 3D Loss
- Learning Single-Image Depth from Videos using Quality Assessment Networks
- An End to End Network Architecture for Fundamental Matrix Estimation
- Variational Monocular Depth Estimation for Reliability Prediction
- Dehazing Cost Volume for Deep Multi-view Stereo in Scattering Media with Airlight and Scattering Coefficient Estimation
- Deep Visual Odometry with Adaptive Memory
- Confidence Inference for Focused Learning in Stereo Matching
- Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias
- A-TVSNet: Aggregated Two-View Stereo Network for Multi-View Stereo Depth Estimation
- SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation
- Projective Manifold Gradient Layer for Deep Rotation Regression
- Attention is All They Need: Exploring the Media Archaeology of the Computer Vision Research Paper
- Exploring Temporal Information for Improved Video Understanding
- 3D Ken Burns Effect from a Single Image
- Flow-Motion and Depth Network for Monocular Stereo and Beyond
- Video Depth Estimation by Fusing Flow-to-Depth Proposals
- Robust Depth Estimation from Auto Bracketed Images
- Y-GAN: A Generative Adversarial Network for Depthmap Estimation from Multi-camera Stereo Images
- Transformer Guided Geometry Model for Flow-Based Unsupervised Visual Odometry
- Adversarial Domain Feature Adaptation for Bronchoscopic Depth Estimation
- Deep Two-View Structure-from-Motion Revisited
- Weakly-Supervised Monocular Depth Estimationwith Resolution-Mismatched Data
- Unsupervised Learning of Depth and Deep Representation for Visual Odometry from Monocular Videos in a Metric Space
- VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals
- Guided Feature Selection for Deep Visual Odometry
- Unstructured Multi-View Depth Estimation Using Mask-Based Multiplane Representation
- Sequential Learning of Visual Tracking and Mapping Using Unsupervised Deep Neural Networks
- Stereo Video Reconstruction Without Explicit Depth Maps for Endoscopic Surgery
- 3D Surface Reconstruction by Pointillism
- Deep Probabilistic Feature-metric Tracking
- FLIVVER: Fly Lobula Inspired Visual Velocity Estimation & Ranging
- Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems
- A Multi-Task Learning Approach for Meal Assessment
- PLG-IN: Pluggable Geometric Consistency Loss with Wasserstein Distance in Monocular Depth Estimation
- Semantics-Driven Unsupervised Learning for Monocular Depth and Ego-Motion Estimation
- 3D Pipe Network Reconstruction Based on Structure from Motion with Incremental Conic Shape Detection and Cylindrical Constraint