Consistent Video Depth Estimation

Abstract

We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects.

SIGGRAPH 2020. Video: https://www.youtube.com/watch?v=5Tia2oblJAg Project page: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/ Code: https://github.com/facebookresearch/consistent_depth

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• FlowNet: Learning Optical Flow with Convolutional Networks
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• SfM-Net: Learning of Structure and Motion from Video
• Synthetic Depth-of-Field with a Single-Camera Mobile Phone
• DeepV2D: Video to Depth with Differentiable Structure from Motion
• BA-Net: Dense Bundle Adjustment Network
• DPSNet: End-to-end Deep Plane Sweep Stereo
• Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
• Normal Assisted Stereo Depth Estimation
• Self-supervised Object Motion and Depth Estimation from Video

Cited by in corpus (3)

• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
• Tangent Space Backpropagation for 3D Transformation Groups
• PLG-IN: Pluggable Geometric Consistency Loss with Wasserstein Distance in Monocular Depth Estimation