PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

Abstract

Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

arXiv commentAccepted to ICLR2021

References in corpus (12)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing
• PU-Net: Point Cloud Upsampling Network
• 3DV: 3D Dynamic Voxel for Action Recognition in Depth Video

Cited by in corpus (8)

• Human Action Recognition from Various Data Modalities: A Review
• GIFS: Neural Implicit Function for General Shape Representation
• StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition
• Unsupervised Motion Representation Learning with Capsule Autoencoders
• SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network
• Domain generalization of 3D semantic segmentation in autonomous driving
• Learning Scene Dynamics from Point Cloud Sequences
• From Multi-View to Hollow-3D: Hallucinated Hollow-3D R-CNN for 3D Object Detection