PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing

Abstract

In this paper, we introduce a Point Recurrent Neural Network (PointRNN) for moving point cloud processing. At each time step, PointRNN takes point coordinates $\mathbf{P}\in {\mathbb{R}}^{n\times 3}$ and point features $\mathbf{X}\in {\mathbb{R}}^{n\times d}$ as input ($n$ and $d$ denote the number of points and the number of feature channels, respectively). The state of PointRNN is composed of point coordinates $\mathbf{P}$ and point states $\mathbf{S}\in {\mathbb{R}}^{n\times d^{\prime }}$ ($d^{\prime }$ denotes the number of state channels). Similarly, the output of PointRNN is composed of $\mathbf{P}$ and new point features $\mathbf{Y}\in {\mathbb{R}}^{n\times d^{\prime \prime }}$ ($d^{\prime \prime }$ denotes the number of new feature channels). Since point clouds are orderless, point features and states from two time steps can not be directly operated. Therefore, a point-based spatiotemporally-local correlation is adopted to aggregate point features and states according to point coordinates. We further propose two variants of PointRNN, i.e., Point Gated Recurrent Unit (PointGRU) and Point Long Short-Term Memory (PointLSTM). We apply PointRNN, PointGRU and PointLSTM to moving point cloud prediction, which aims to predict the future trajectories of points in a set given their history movements. Experimental results show that PointRNN, PointGRU and PointLSTM are able to produce correct predictions on both synthetic and real-world datasets, demonstrating their ability to model point cloud sequences. The code has been released at \url{https://github.com/hehefan/PointRNN}.

technical report

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Learning of Video Representations using LSTMs
• Learning Representations and Generative Models for 3D Point Clouds
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Argoverse: 3D Tracking and Forecasting with Rich Maps

Cited by in corpus (13)

• PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
• Deep Learning for 3D Point Clouds: A Survey
• Deep Learning for 3D Point Cloud Understanding: A Survey
• Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion
• Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks
• An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
• RAFT-3D: Scene Flow using Rigid-Motion Embeddings
• Spatial-Temporal Transformer for 3D Point Cloud Sequences
• Temporal LiDAR Frame Prediction for Autonomous Driving
• MoNet: Motion-based Point Cloud Prediction Network
• Learning Scene Dynamics from Point Cloud Sequences
• 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation
• Human Segmentation with Dynamic LiDAR Data