Self-Supervised Learning of Depth and Camera Motion from 360° Videos

Abstract

As 360° cameras become prevalent in many autonomous systems (e.g., self-driving cars and drones), efficient 360° perception becomes more and more important. We propose a novel self-supervised learning approach for predicting the omnidirectional depth and camera motion from a 360° video. In particular, starting from the SfMLearner, which is designed for cameras with normal field-of-view, we introduce three key features to process 360° images efficiently. Firstly, we convert each image from equirectangular projection to cubic projection in order to avoid image distortion. In each network layer, we use Cube Padding (CP), which pads intermediate features from adjacent faces, to avoid image boundaries. Secondly, we propose a novel "spherical" photometric consistency constraint on the whole viewing sphere. In this way, no pixel will be projected outside the image boundary which typically happens in images with normal field-of-view. Finally, rather than naively estimating six independent camera motions (i.e., naively applying SfM-Learner to each face on a cube), we propose a novel camera pose consistency loss to ensure the estimated camera motions reaching consensus. To train and evaluate our approach, we collect a new PanoSUNCG dataset containing a large amount of 360° videos with groundtruth depth and camera motion. Our approach achieves state-of-the-art depth prediction and camera motion estimation on PanoSUNCG with faster inference speed comparing to equirectangular. In real-world indoor videos, our approach can also achieve qualitatively reasonable depth prediction by acquiring model pre-trained on PanoSUNCG.

arXiv commentACCV 2018 Oral

References in corpus (8)

• SfM-Net: Learning of Structure and Motion from Video
• Learning Spherical Convolution for Fast Features from 360° Imagery
• DeepStereo: Learning to Predict New Views from the World's Imagery
• CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction
• Learning Depth from Monocular Videos using Direct Methods
• Omnidirectional CNN for Visual Place Recognition and Navigation
• Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video
• 3D Visual Perception for Self-Driving Cars using a Multi-Camera System: Calibration, Mapping, Localization, and Obstacle Detection

Cited by in corpus (6)

• Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-supervised Learning
• LayoutMP3D: Layout Annotation of Matterport3D
• Spherical View Synthesis for Self-Supervised 360 Depth Estimation
• 360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume
• LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering
• $360^o$ Surface Regression with a Hyper-Sphere Loss