Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

Abstract

It remains a challenge to efficiently extract spatialtemporal information from skeleton sequences for 3D human action recognition. Although most recent action recognition methods are based on Recurrent Neural Networks which present outstanding performance, one of the shortcomings of these methods is the tendency to overemphasize the temporal information. Since 3D convolutional neural network(3D CNN) is a powerful tool to simultaneously learn features from both spatial and temporal dimensions through capturing the correlations between three dimensional signals, this paper proposes a novel two-stream model using 3D CNN. To our best knowledge, this is the first application of 3D CNN in skeleton-based action recognition. Our method consists of three stages. First, skeleton joints are mapped into a 3D coordinate space and then encoding the spatial and temporal information, respectively. Second, 3D CNN models are seperately adopted to extract deep features from two streams. Third, to enhance the ability of deep features to capture global relationships, we extend every stream into multitemporal version. Extensive experiments on the SmartHome dataset and the large-scale NTU RGB-D dataset demonstrate that our method outperforms most of RNN-based methods, which verify the complementary property between spatial and temporal information and the robustness to noise.

5 pages, 6 figures, 3 tabels

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• 3D Fully Convolutional Network for Vehicle Detection in Point Cloud
• Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition

Cited by in corpus (11)

• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
• Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition
• Feedback Graph Convolutional Network for Skeleton-based Action Recognition
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• STAR: Sparse Transformer-based Action Recognition
• Relational Network for Skeleton-Based Action Recognition
• Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition
• Spiking Two-Stream Methods with Unsupervised STDP-based Learning for Action Recognition
• Spatio-Temporal Dual Affine Differential Invariant for Skeleton-based Action Recognition
• Improving Skeleton-based Action Recognitionwith Robust Spatial and Temporal Features
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition