Orientation-aware Semantic Segmentation on Icosahedron Spheres
arXiv:1907.12849
Abstract
We address semantic segmentation on omnidirectional images, to leverage a holistic understanding of the surrounding scene for applications like autonomous driving systems. For the spherical domain, several methods recently adopt an icosahedron mesh, but systems are typically rotation invariant or require significant memory and parameters, thus enabling execution only at very low resolutions. In our work, we propose an orientation-aware CNN framework for the icosahedron mesh. Our representation allows for fast network operations, as our design simplifies to standard network operations of classical CNNs, but under consideration of north-aligned kernel convolutions for features on the sphere. We implement our representation and demonstrate its memory efficiency up-to a level-8 resolution mesh (equivalent to 640 x 1024 equirectangular images). Finally, since our kernels operate on the tangent of the sphere, standard feature weights, pretrained on perspective data, can be directly transferred with only small need for weight refinement. In our evaluation our orientation-aware CNN becomes a new state of the art for the recent 2D3DS dataset, and our Omni-SYNTHIA version of SYNTHIA. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.
arXiv comment9 pages, accepted to iccv 2019
References in corpus (8)
- Geometric deep learning: going beyond Euclidean data
- Joint 2D-3D-Semantic Data for Indoor Scene Understanding
- 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- Spherical CNNs
- Learning Spherical Convolution for Fast Features from 360° Imagery
- Spherical CNNs on Unstructured Grids
- SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images