Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition

Abstract

One essential problem in skeleton-based action recognition is how to extract discriminative features over all skeleton joints. However, the complexity of the State-Of-The-Art (SOTA) models of this task tends to be exceedingly sophisticated and over-parameterized, where the low efficiency in model training and inference has obstructed the development in the field, especially for large-scale action datasets. In this work, we propose an efficient but strong baseline based on Graph Convolutional Network (GCN), where three main improvements are aggregated, i.e., early fused Multiple Input Branches (MIB), Residual GCN (ResGCN) with bottleneck structure and Part-wise Attention (PartAtt) block. Firstly, an MIB is designed to enrich informative skeleton features and remain compact representations at an early fusion stage. Then, inspired by the success of the ResNet architecture in Convolutional Neural Network (CNN), a ResGCN module is introduced in GCN to alleviate computational costs and reduce learning difficulties in model training while maintain the model accuracy. Finally, a PartAtt block is proposed to discover the most essential body parts over a whole action sequence and obtain more explainable representations for different skeleton action sequences. Extensive experiments on two large-scale datasets, i.e., NTU RGB+D 60 and 120, validate that the proposed baseline slightly outperforms other SOTA models and meanwhile requires much fewer parameters during training and inference procedures, e.g., at most 34 times less than DGNN, which is one of the best SOTA methods.

arXiv commentAccepted by ACM MultiMedia 2020, 9 pages, 4 figures, 5 tables

References in corpus (8)

• Semi-Supervised Classification with Graph Convolutional Networks
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• Part-based Graph Convolutional Network for Action Recognition
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Make Skeleton-based Action Recognition Model Smaller, Faster and Better
• Vertex Feature Encoding and Hierarchical Temporal Modeling in a Spatial-Temporal Graph Convolutional Network for Action Recognition

Cited by in corpus (15)

• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• Two-person Graph Convolutional Network for Skeleton-based Human Interaction Recognition
• Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble
• IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition
• Skeleton Aware Multi-modal Sign Language Recognition
• STAR: Sparse Transformer-based Action Recognition
• SMART-Vision: Survey of Modern Action Recognition Techniques in Vision
• Word-level Sign Language Recognition with Multi-stream Neural Networks Focusing on Local Regions and Skeletal Information
• Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition
• Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition
• Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition
• Action Recognition in Real-World Ambient Assisted Living Environment
• LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition
• NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions