Learning Spherical Convolution for Fast Features from 360° Imagery

Abstract

While 360° cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial. Convolutional neural networks (CNNs) trained on images from perspective cameras yield "flat" filters, yet 360° images cannot be projected to a single plane without significant distortion. A naive solution that repeatedly projects the viewing sphere to all tangent planes is accurate, but much too computationally intensive for real problems. We propose to learn a spherical convolutional network that translates a planar CNN to process 360° imagery directly in its equirectangular projection. Our approach learns to reproduce the flat filter outputs on 360° data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient feature extraction for 360° images and video, and 2) the ability to leverage powerful pre-trained networks researchers have carefully honed (together with massive labeled image training sets) for perspective images. We validate our approach compared to several alternative methods in terms of both raw CNN output accuracy as well as applying a state-of-the-art "flat" object detector to 360° data. Our method yields the most accurate results while saving orders of magnitude in computation versus the existing exact reprojection solution.

NIPS 2017

Cited by in corpus (36)

• DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications
• Improving prediction of the terrestrial water storage anomalies during the GRACE and GRACE-FO gap with Bayesian convolutional neural networks
• Cortical surface registration using unsupervised learning
• OSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression
• Convolutions on Spherical Images
• DeepSphere: a graph-based spherical CNN
• Kernel Transformer Networks for Compact Spherical Convolution
• DeepSphere: towards an equivariant graph-based spherical CNN
• Self-Supervised Learning of Depth and Camera Motion from 360° Videos
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images
• LayoutMP3D: Layout Annotation of Matterport3D
• Spherical View Synthesis for Self-Supervised 360 Depth Estimation
• 360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume
• HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features
• Orientation-aware Semantic Segmentation on Icosahedron Spheres
• 360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images
• Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality
• Visual Question Answering on 360° Images
• ASOD60K: An Audio-Induced Salient Object Detection Dataset for Panoramic Videos
• LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering
• Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network
• Mapped Convolutions
• Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos
• Spherical Transformer: Adapting Spherical Signal to CNNs
• Spherical Image Generation from a Single Normal Field of View Image by Considering Scene Symmetry
• ODE-CNN: Omnidirectional Depth Extension Networks
• Reprojection R-CNN: A Fast and Accurate Object Detector for 360° Images
• Spherical Convolutional Neural Networks: Stability to Perturbations in SO(3)
• Equivariant Networks for Pixelized Spheres
• PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers
• Distortion-Adaptive Grape Bunch Counting for Omnidirectional Images
• Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360° Panoramic Imagery
• $360^o$ Surface Regression with a Hyper-Sphere Loss
• Revisiting Optical Flow Estimation in 360 Videos
• Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset