GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation

Abstract

In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the "depth-to-normal" module exploits the least square solution of estimating surface normals from depth to improve their quality, while the "normal-to-depth" module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with strong 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve the quality of 3D reconstruction and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high-quality 3D surface normals. This is a more natural metric for many 3D application domains. Our experiments on NYUD-V2 and KITTI datasets verify that GeoNet++ produces fine boundary details, and the predicted depth can be used to reconstruct high-quality 3D surfaces. Code has been made publicly available.

TPAMI 2020. Code available: https://github.com/xjqi/GeoNet

References in corpus (6)

• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• ParseNet: Looking Wider to See Better
• Depth Extraction from Video Using Non-parametric Sampling
• PixelNet: Representation of the pixels, by the pixels, and for the pixels

Cited by in corpus (4)

• Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
• IRS: A Large Naturalistic Indoor Robotics Stereo Dataset to Train Deep Models for Disparity and Surface Normal Estimation
• Normal Transformer: Extracting Surface Geometry from LiDAR Points Enhanced by Visual Semantics
• S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation