SurfConv: Bridging 3D and 2D Convolution for RGBD Images
arXiv:1812.01519
Abstract
We tackle the problem of using 3D information in convolutional neural networks for down-stream recognition tasks. Using depth as an additional channel alongside the RGB input has the scale variance problem present in image convolution based approaches. On the other hand, 3D convolution wastes a large amount of memory on mostly unoccupied 3D space, which consists of only the surface visible to the sensor. Instead, we propose SurfConv, which "slides" compact 2D filters along the visible 3D surface. SurfConv is formulated as a simple depth-aware multi-scale 2D convolution, through a new Data-Driven Depth Discretization (D4) scheme. We demonstrate the effectiveness of our method on indoor and outdoor 3D semantic segmentation datasets. Our method achieves state-of-the-art performance with less than 30% parameters used by the 3D convolution-based approaches.
arXiv commentPublished at CVPR 2018
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Fully Convolutional Networks for Semantic Segmentation
- Object Detectors Emerge in Deep Scene CNNs
- ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
- Feature Pyramid Networks for Object Detection
- Deformable Convolutional Networks
- Pyramid Scene Parsing Network
- Vehicle Detection from 3D Lidar Using Fully Convolutional Network
- Learning shape correspondence with anisotropic convolutional neural networks
- Attention to Scale: Scale-aware Semantic Image Segmentation
- 3D ShapeNets: A Deep Representation for Volumetric Shapes
- Multi-View 3D Object Detection Network for Autonomous Driving
- 3D Object Proposals using Stereo Imagery for Accurate Object Class Detection
- Semantic Scene Completion from a Single Depth Image
- Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
- OctNet: Learning Deep 3D Representations at High Resolutions
- Scale-Aware Face Detection
- Cross Modal Distillation for Supervision Transfer
- TorontoCity: Seeing the World with a Million Eyes
- 3D Shape Segmentation with Projective Convolutional Networks
- DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding