Infrared and 3D skeleton feature fusion for RGB-D action recognition
arXiv:2002.12886 · doi:10.1109/ACCESS.2020.3023599
Abstract
A challenge of skeleton-based action recognition is the difficulty to classify actions with similar motions and object-related actions. Visual clues from other streams help in that regard. RGB data are sensible to illumination conditions, thus unusable in the dark. To alleviate this issue and still benefit from a visual stream, we propose a modular network (FUSION) combining skeleton and infrared data. A 2D convolutional neural network (CNN) is used as a pose module to extract features from skeleton data. A 3D CNN is used as an infrared module to extract visual cues from videos. Both feature vectors are then concatenated and exploited conjointly using a multilayer perceptron (MLP). Skeleton data also condition the infrared videos, providing a crop around the performing subjects and thus virtually focusing the attention of the infrared module. Ablation studies show that using pre-trained networks on other large scale datasets as our modules and data augmentation yield considerable improvements on the action classification accuracy. The strong contribution of our cropping strategy is also demonstrated. We evaluate our method on the NTU RGB+D dataset, the largest dataset for human action recognition from depth cameras, and report state-of-the-art performances.
arXiv comment11 pages, 5 figures, submitted to IEEE Access
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Recurrent Models of Visual Attention
- A New Representation of Skeleton Sequences for 3D Action Recognition
- An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
- How Powerful are Graph Neural Networks?
- Action Recognition using Visual Attention
- Describing Multimedia Content using Attention-based Encoder--Decoder Networks
- Skeleton-based Action Recognition with Convolutional Neural Networks
- PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
- Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
- Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn
- Cooperative Training of Deep Aggregation Networks for RGB-D Action Recognition