Multi-View 3D Object Detection Network for Autonomous Driving
arXiv:1611.07759
Abstract
This paper aims at high-accuracy 3D object detection in autonomous driving scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework that takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes. We encode the sparse 3D point cloud with a compact multi-view representation. The network is composed of two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion. The proposal network generates 3D candidate boxes efficiently from the bird's eye view representation of 3D point cloud. We design a deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths. Experiments on the challenging KITTI benchmark show that our approach outperforms the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection. In addition, for 2D detection, our approach obtains 10.3% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.
arXiv commentTo appear in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- What makes for effective detection proposals?
- FractalNet: Ultra-Deep Neural Networks without Residuals
- Vehicle Detection from 3D Lidar Using Fully Convolutional Network
- FusionNet: 3D Object Classification Using Multiple Data Representations
- Deeply-Fused Nets
- 3D Object Proposals using Stereo Imagery for Accurate Object Class Detection
- Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
- 3D Fully Convolutional Network for Vehicle Detection in Point Cloud
Cited by in corpus (27)
- SINet: A Scale-insensitive Convolutional Neural Network for Fast Vehicle Detection
- Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
- End-to-end Learning of Multi-sensor 3D Tracking by Detection
- Orthographic Feature Transform for Monocular 3D Object Detection
- 3DSSD: Point-based 3D Single Stage Object Detector
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
- SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
- SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation
- Fast LIDAR-based Road Detection Using Fully Convolutional Neural Networks
- In Defense of Classical Image Processing: Fast Depth Completion on the CPU
- A survey of Object Classification and Detection based on 2D/3D data
- Driving Datasets Literature Review
- Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
- Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
- Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection
- Efficient Deep Neural Networks
- 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction
- SurfConv: Bridging 3D and 2D Convolution for RGBD Images
- Frustum VoxNet for 3D object detection from RGB-D or Depth images
- Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
- PointNet on FPGA for Real-Time LiDAR Point Cloud Processing
- Joint Spatial-Temporal Optimization for Stereo 3D Object Tracking
- A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection
- Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration