DeepSFM: Structure From Motion Via Deep Bundle Adjustment

Abstract

Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. However, existing methods usually assume accurate camera poses either from GT or other methods, which is unrealistic in practice. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running to improve both. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.

arXiv commentAccepted by ECCV2020(Oral)

References in corpus (8)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• DeepV2D: Video to Depth with Differentiable Structure from Motion
• BA-Net: Dense Bundle Adjustment Network
• DPSNet: End-to-end Deep Plane Sweep Stereo