Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition

Abstract

Skeleton-based human action recognition has attracted great interest thanks to the easy accessibility of the human skeleton data. Recently, there is a trend of using very deep feedforward neural networks to model the 3D coordinates of joints without considering the computational efficiency. In this paper, we propose a simple yet effective semantics-guided neural network (SGN) for skeleton-based action recognition. We explicitly introduce the high level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability. In addition, we exploit the relationship of joints hierarchically through two modules, i.e., a joint-level module for modeling the correlations of joints in the same frame and a framelevel module for modeling the dependencies of frames by taking the joints in the same frame as a whole. A strong baseline is proposed to facilitate the study of this field. With an order of magnitude smaller model size than most previous works, SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets. The source code is available at https://github.com/microsoft/SGN.

arXiv commentAccepted by CVPR2020. The source code is available at https://github.com/microsoft/SGN

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Semi-Supervised Classification with Graph Convolutional Networks
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• The Microsoft 2017 Conversational Speech Recognition System
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data
• Learning Deep Bilinear Transformation for Fine-grained Image Representation
• Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning
• Adding Attentiveness to the Neurons in Recurrent Neural Networks

Cited by in corpus (2)

• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition