S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation
arXiv:2104.00877
Abstract
Human can infer the 3D geometry of a scene from a sketch instead of a realistic image, which indicates that the spatial structure plays a fundamental role in understanding the depth of scenes. We are the first to explore the learning of a depth-specific structural representation, which captures the essential feature for depth estimation and ignores irrelevant style information. Our S2R-DepthNet (Synthetic to Real DepthNet) can be well generalized to unseen real-world data directly even though it is only trained on synthetic data. S2R-DepthNet consists of: a) a Structure Extraction (STE) module which extracts a domaininvariant structural representation from an image by disentangling the image into domain-invariant structure and domain-specific style components, b) a Depth-specific Attention (DSA) module, which learns task-specific knowledge to suppress depth-irrelevant structures for better depth estimation and generalization, and c) a depth prediction module (DP) to predict depth from the depth-specific representation. Without access of any real-world images, our method even outperforms the state-of-the-art unsupervised domain adaptation methods which use real-world images of the target domain for training. In addition, when using a small amount of labeled real-world data, we achieve the state-ofthe-art performance under the semi-supervised setting. The code and trained models are available at https://github.com/microsoft/S2R-DepthNet.
arXiv commentAccepted by CVPR2021(oral)
References in corpus (17)
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
- Unsupervised Domain Adaptation with Residual Transfer Networks
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Domain Generalization via Invariant Feature Representation
- Generalizing Across Domains via Cross-Gradient Training
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
- Virtual Worlds as Proxy for Multi-Object Tracking Analysis
- Unsupervised Learning of Depth and Ego-Motion from Video
- Deep Ordinal Regression Network for Monocular Depth Estimation
- GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
- DPSNet: End-to-end Deep Plane Sweep Stereo
- GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation
- Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
- Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation
- Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation
- Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera