LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
arXiv:2004.01389
Abstract
Existing LiDAR-based 3D object detectors usually focus on the single-frame detection, while ignoring the spatiotemporal information in consecutive point cloud frames. In this paper, we propose an end-to-end online 3D video object detector that operates on point cloud sequences. The proposed model comprises a spatial feature encoding component and a spatiotemporal feature aggregation component. In the former component, a novel Pillar Message Passing Network (PMPNet) is proposed to encode each discrete point cloud frame. It adaptively collects information for a pillar node from its neighbors by iterative message passing, which effectively enlarges the receptive field of the pillar feature. In the latter component, we propose an Attentive Spatiotemporal Transformer GRU (AST-GRU) to aggregate the spatiotemporal information, which enhances the conventional ConvGRU with an attentive memory gating mechanism. AST-GRU contains a Spatial Transformer Attention (STA) module and a Temporal Transformer Attention (TTA) module, which can emphasize the foreground objects and align the dynamic objects, respectively. Experimental results demonstrate that the proposed 3D video object detector achieves state-of-the-art performance on the large-scale nuScenes benchmark.
Accepted to CVPR 2020. Code: https://github.com/yinjunbo/3DVID
References in corpus (10)
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Deep Convolutional Networks on Graph-Structured Data
- Interaction Networks for Learning about Objects, Relations and Physics
- Deep Continuous Fusion for Multi-Sensor 3D Object Detection
- Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
- An Empirical Study of Spatial Attention Mechanisms in Deep Networks
- STD: Sparse-to-Dense 3D Object Detector for Point Cloud
- GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving
- Infinitely Wide Graph Convolutional Networks: Semi-supervised Learning via Gaussian Processes