Skeleton-based Action Recognition with Convolutional Neural Networks

Abstract

Current state-of-the-art approaches to skeleton-based action recognition are mostly based on recurrent neural networks (RNN). In this paper, we propose a novel convolutional neural networks (CNN) based framework for both action classification and detection. Raw skeleton coordinates as well as skeleton motion are fed directly into CNN for label prediction. A novel skeleton transformer module is designed to rearrange and select important skeleton joints automatically. With a simple 7-layer network, we obtain 89.3% accuracy on validation set of the NTU RGB+D dataset. For action detection in untrimmed videos, we develop a window proposal network to extract temporal segment proposals, which are further classified within the same network. On the recent PKU-MMD dataset, we achieve 93.7% mAP, surpassing the baseline by a large margin.

ICMEW 2017

Cited by in corpus (8)

• Human Action Recognition from Various Data Modalities: A Review
• Exploiting deep residual networks for human action recognition from skeletal data
• Dynamic Gesture Recognition by Using CNNs and Star RGB: a Temporal Information Condensation
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• A Central Difference Graph Convolutional Operator for Skeleton-Based Action Recognition
• Subspace Clustering for Action Recognition with Covariance Representations and Temporal Pruning
• When Kernel Methods meet Feature Learning: Log-Covariance Network for Action Recognition from Skeletal Data
• Real-time Recognition of Human Interactions from a Single RGB-D Camera for Socially-Aware Robot Navigation