Spherical CNNs

Abstract

Convolutional Neural Networks (CNNs) have become the method of choice for learning problems involving 2D planar images. However, a number of problems of recent interest have created a demand for models that can analyze spherical images. Examples include omnidirectional vision for drones, robots, and autonomous cars, molecular regression problems, and global weather and climate modelling. A naive application of convolutional networks to a planar projection of the spherical signal is destined to fail, because the space-varying distortions introduced by such a projection will make translational weight sharing ineffective. In this paper we introduce the building blocks for constructing spherical CNNs. We propose a definition for the spherical cross-correlation that is both expressive and rotation-equivariant. The spherical correlation satisfies a generalized Fourier theorem, which allows us to compute it efficiently using a generalized (non-commutative) Fast Fourier Transform (FFT) algorithm. We demonstrate the computational efficiency, numerical accuracy, and effectiveness of spherical CNNs applied to 3D model recognition and atomization energy regression.

Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018

Cited by in corpus (54)

• Machine learning and the physical sciences
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
• Analog forecasting of extreme-causing weather patterns using deep learning
• End-to-end Symmetry Preserving Inter-atomic Potential Energy Model for Finite and Extended Systems
• A Rotation-Invariant Framework for Deep Point Cloud Analysis
• 3D G-CNNs for Pulmonary Nodule Detection
• Three-dimensional microstructure generation using generative adversarial neural networks in the context of continuum micromechanics
• Efficient Modelling of Trivializing Maps for Lattice ${\varphi }^4$ Theory Using Normalizing Flows: A First Look at Scalability
• Cortical surface registration using unsupervised learning
• Full-sky Cosmic Microwave Background Foreground Cleaning Using Machine Learning
• Intertwiners between Induced Representations (with Applications to the Theory of Equivariant Neural Networks)
• Attentive Group Equivariant Convolutional Networks
• Rotation Invariant Point Cloud Classification: Where Local Geometry Meets Global Topology
• A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels
• Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales
• Suppression of Cosmic Muon Spallation Backgrounds in Liquid Scintillator Detectors Using Convolutional Neural Networks
• You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors
• Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution
• Improved Person Detection on Omnidirectional Images with Non-maxima Suppression
• Kernel Transformer Networks for Compact Spherical Convolution
• B-Spline CNNs on Lie Groups
• 3D-Rotation-Equivariant Quaternion Neural Networks
• Scaling-Translation-Equivariant Networks with Decomposed Convolutional Filters
• Frame Averaging for Invariant and Equivariant Network Design
• Spin-Weighted Spherical CNNs
• SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images
• View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions
• LayoutMP3D: Layout Annotation of Matterport3D
• Orientation-aware Semantic Segmentation on Icosahedron Spheres
• HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features
• Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs
• Disentangling by Subspace Diffusion
• Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes
• Geometric Wavelet Scattering Networks on Compact Riemannian Manifolds
• Quaternion Product Units for Deep Learning on 3D Rotation Groups
• Learnable Gabor modulated complex-valued networks for orientation robustness
• Isometric Transformation Invariant and Equivariant Graph Convolutional Networks
• Learning Compressed Transforms with Low Displacement Rank
• Spherical Transformer: Adapting Spherical Signal to CNNs
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• Reprojection R-CNN: A Fast and Accurate Object Detector for 360° Images
• Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space
• DeepConsensus: using the consensus of features from multiple layers to attain robust image classification
• Equivariant Deep Dynamical Model for Motion Prediction
• Spherical Convolutional Neural Networks: Stability to Perturbations in SO(3)
• Efficient Generalized Spherical CNNs
• Quantised Transforming Auto-Encoders: Achieving Equivariance to Arbitrary Transformations in Deep Networks
• Autoequivariant Network Search via Group Decomposition
• Equivariant Manifold Flows
• Fast Markov Chain Monte Carlo Algorithms via Lie Groups
• Geometry-Aware Self-Training for Unsupervised Domain Adaptationon Object Point Clouds
• Self-Refining Deep Symmetry Enhanced Network for Rain Removal