Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition

Abstract

Spatial-temporal graphs have been widely used by skeleton-based action recognition algorithms to model human action dynamics. To capture robust movement patterns from these graphs, long-range and multi-scale context aggregation and spatial-temporal dependency modeling are critical aspects of a powerful feature extractor. However, existing methods have limitations in achieving (1) unbiased long-range joint relationship modeling under multi-scale operators and (2) unobstructed cross-spacetime information flow for capturing complex spatial-temporal dependencies. In this work, we present (1) a simple method to disentangle multi-scale graph convolutions and (2) a unified spatial-temporal graph convolutional operator named G3D. The proposed multi-scale aggregation scheme disentangles the importance of nodes in different neighborhoods for effective long-range modeling. The proposed G3D module leverages dense cross-spacetime edges as skip connections for direct information propagation across the spatial-temporal graph. By coupling these proposals, we develop a powerful feature extractor named MS-G3D based on which our model outperforms previous state-of-the-art methods on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.

arXiv commentCVPR 2020

References in corpus (11)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• The Kinetics Human Action Video Dataset
• Spectral Networks and Locally Connected Networks on Graphs
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Multi-Scale Context Aggregation by Dilated Convolutions
• Deep Convolutional Networks on Graph-Structured Data
• Simplifying Graph Convolutional Networks
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• LanczosNet: Multi-Scale Deep Graph Convolutional Networks

Cited by in corpus (5)

• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition
• Towards Understanding the Adversarial Vulnerability of Skeleton-based Action Recognition
• Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition
• Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective
• What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition