A New Representation of Skeleton Sequences for 3D Action Recognition

Abstract

This paper presents a new method for 3D action recognition with skeleton sequences (i.e., 3D trajectories of human skeleton joints). The proposed method first transforms each skeleton sequence into three clips each consisting of several frames for spatial temporal feature learning using deep neural networks. Each clip is generated from one channel of the cylindrical coordinates of the skeleton sequence. Each frame of the generated clips represents the temporal information of the entire skeleton sequence, and incorporates one particular spatial relationship between the joints. The entire clips include multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We propose to use deep convolutional neural networks to learn long-term temporal information of the skeleton sequence from the frames of the generated clips, and then use a Multi-Task Learning Network (MTLN) to jointly process all frames of the generated clips in parallel to incorporate spatial structural information for action recognition. Experimental results clearly show the effectiveness of the proposed new representation and feature learning method for 3D action recognition.

CVPR 2017

Cited by in corpus (23)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Human Action Recognition from Various Data Modalities: A Review
• Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks
• Semantic Graph Convolutional Networks for 3D Human Pose Regression
• A Comparative Review of Recent Kinect-based Action Recognition Algorithms
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• MS${}^2$L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks
• Ensemble One-dimensional Convolution Neural Networks for Skeleton-based Action Recognition
• Synthetic Humans for Action Recognition from Unseen Viewpoints
• Tensor Representations for Action Recognition
• On-field player workload exposure and knee injury risk monitoring via deep learning
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• Skeleton-based Action Recognition of People Handling Objects
• Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction
• Human Action Performance using Deep Neuro-Fuzzy Recurrent Attention Model
• DanceIt: Music-inspired Dancing Video Synthesis
• Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton based Action Recognition
• Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
• Subspace Clustering for Action Recognition with Covariance Representations and Temporal Pruning
• SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network
• Local Spherical Harmonics Improve Skeleton-Based Hand Action Recognition