An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data

Abstract

Human action recognition is an important task in computer vision. Extracting discriminative spatial and temporal features to model the spatial and temporal evolutions of different actions plays a key role in accomplishing this task. In this work, we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames. Furthermore, to ensure effective training of the network, we propose a regularized cross-entropy loss to drive the model learning process and develop a joint training strategy accordingly. Experimental results demonstrate the effectiveness of the proposed model,both on the small human action recognition data set of SBU and the currently largest NTU dataset.

References in corpus (7)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Recurrent Neural Network Regularization
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
• Recurrent Mixture Density Network for Spatiotemporal Visual Attention

Cited by in corpus (95)

• Attention Mechanisms in Computer Vision: A Survey
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition
• Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognition
• A Deep Learning Framework for Assessing Physical Rehabilitation Exercises
• Attentional Pooling for Action Recognition
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• A Review of Computational Approaches for Evaluation of Rehabilitation Exercises
• CCSNet: a deep learning modeling suite for CO${}_2$ storage
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition
• Exploiting deep residual networks for human action recognition from skeletal data
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• Action-Attending Graphic Neural Network
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things
• Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Deep Independently Recurrent Neural Network (IndRNN)
• Human Action Performance using Deep Neuro-Fuzzy Recurrent Attention Model
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning
• Make Skeleton-based Action Recognition Model Smaller, Faster and Better
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
• Complex Sequential Understanding through the Awareness of Spatial and Temporal Concepts
• View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition
• UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition
• Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition
• Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
• Feedback Graph Convolutional Network for Skeleton-based Action Recognition
• Detailed 2D-3D Joint Representation for Human-Object Interaction
• HyperST-Net: Hypernetworks for Spatio-Temporal Forecasting
• Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
• GTA: Global Temporal Attention for Video Action Understanding
• Learning Group Activities from Skeletons without Individual Action Labels
• Optimized Skeleton-based Action Recognition via Sparsified Graph Regression
• Relational Network for Skeleton-Based Action Recognition
• MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction
• Understanding More about Human and Machine Attention in Deep Neural Networks
• Focusing and Diffusion: Bidirectional Attentive Graph Convolutional Networks for Skeleton-based Action Recognition
• Learning Representative Temporal Features for Action Recognition
• Interpretable Spatio-temporal Attention for Video Action Recognition
• Skeleton Boxes: Solving skeleton based action detection with a single deep convolutional neural network
• Video Super-resolution with Temporal Group Attention
• Collaborative Attention Mechanism for Multi-View Action Recognition
• ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
• A Two-stream Neural Network for Pose-based Hand Gesture Recognition
• CTCModel: a Keras Model for Connectionist Temporal Classification
• Adding Attentiveness to the Neurons in Recurrent Neural Networks
• Understanding the Robustness of Skeleton-based Action Recognition under Adversarial Attack
• HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Pose-Based Two-Stream Relational Networks for Action Recognition in Videos
• Group-Skeleton-Based Human Action Recognition in Complex Events
• Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks
• BASAR:Black-box Attack on Skeletal Action Recognition
• SMART: Skeletal Motion Action Recognition aTtack
• Pose Refinement Graph Convolutional Network for Skeleton-based Action Recognition
• Skeleton based Activity Recognition by Fusing Part-wise Spatio-temporal and Attention Driven Residues
• Delving into 3D Action Anticipation from Streaming Videos
• Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach
• Action Recognition with Coarse-to-Fine Deep Feature Integration and Asynchronous Fusion
• Human Action Recognition with Multi-Laplacian Graph Convolutional Networks
• Mix Dimension in Poincaré Geometry for 3D Skeleton-based Action Recognition
• Reasoning About Human-Object Interactions Through Dual Attention Networks
• Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Predictively Encoded Graph Convolutional Network for Noise-Robust Skeleton-based Action Recognition
• TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation
• Temporal Extension Module for Skeleton-Based Action Recognition
• Cross-modal knowledge distillation for action recognition
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
• Selective Spatio-Temporal Aggregation Based Pose Refinement System: Towards Understanding Human Activities in Real-World Videos
• What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition
• Action Recognition with Spatio-Temporal Visual Attention on Skeleton Image Sequences
• An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos
• An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal
• Improving Skeleton-based Action Recognitionwith Robust Spatial and Temporal Features
• GID-Net: Detecting Human-Object Interaction with Global and Instance Dependency
• STS Classification with Dual-stream CNN
• Learning Various Length Dependence by Dual Recurrent Neural Networks
• SAR-NAS: Skeleton-based Action Recognition via Neural Architecture Searching
• Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition
• Totally Deep Support Vector Machines
• Learning Coupled Spatial-temporal Attention for Skeleton-based Action Recognition
• Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition
• Video-Based Convolutional Attention for Person Re-Identification
• Skeleton-based Activity Recognition with Local Order Preserving Match of Linear Patches
• Object Properties Inferring from and Transfer for Human Interaction Motions
• Learning Object Permanence from Video
• Action Recognition with Kernel-based Graph Convolutional Networks
• DeepActsNet: Spatial and Motion features from Face, Hands, and Body Combined with Convolutional and Graph Networks for Improved Action Recognition
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition
• Modality Compensation Network: Cross-Modal Adaptation for Action Recognition
• Interpretable 3D Human Action Analysis with Temporal Convolutional Networks
• Attention-Driven Body Pose Encoding for Human Activity Recognition