ODE-CNN: Omnidirectional Depth Extension Networks

Abstract

Omnidirectional 360° camera proliferates rapidly for autonomous robots since it significantly enhances the perception ability by widening the field of view(FoV). However, corresponding 360° depth sensors, which are also critical for the perception system, are still difficult or expensive to have. In this paper, we propose a low-cost 3D sensing system that combines an omnidirectional camera with a calibrated projective depth camera, where the depth from the limited FoV can be automatically extended to the rest of the recorded omnidirectional image. To accurately recover the missing depths, we design an omnidirectional depth extension convolutional neural network(ODE-CNN), in which a spherical feature transform layer(SFTL) is embedded at the end of feature encoding layers, and a deformable convolutional spatial propagation network(D-CSPN) is appended at the end of feature decoding layers. The former resamples the neighborhood of each pixel in the omnidirectional coordination to the projective coordination, which reduces the difficulty of feature learning, and the later automatically finds a proper context to well align the structures in the estimated depths via CNN w.r.t. the reference image, which significantly improves the visual quality. Finally, we demonstrate the effectiveness of proposed ODE-CNN over the popular 360D dataset and show that ODE-CNN significantly outperforms (relatively 33% reduction in-depth error) other state-of-the-art (SoTA) methods.

Accepted by ICRA 2020, 7 pages, 5 figures

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras
• Learning Spherical Convolution for Fast Features from 360° Imagery
• Convolutional Networks for Spherical Signals
• Learning Depth with Convolutional Spatial Propagation Network
• Mapped Convolutions

Cited by in corpus (2)

• Distortion-Tolerant Monocular Depth Estimation On Omnidirectional Images Using Dual-cubemap
• LEAD: LiDAR Extender for Autonomous Driving