Differential Recurrent Neural Networks for Action Recognition

Abstract

The long short-term memory (LSTM) neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any sequential time-series data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics of various actions. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). We demonstrate the effectiveness of the proposed model by automatically recognizing actions from the real-world 2D and 3D human action datasets. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.

References in corpus (1)

• DRAW: A Recurrent Neural Network For Image Generation

Cited by in corpus (34)

• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Recent Advances in Recurrent Neural Networks
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Exploiting deep residual networks for human action recognition from skeletal data
• Artificial Intelligence and its Role in Near Future
• Learning to Represent Edits
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks
• A Survey of Deep Learning Techniques for Mobile Robot Applications
• VRUNet: Multi-Task Learning Model for Intent Prediction of Vulnerable Road Users
• ActionXPose: A Novel 2D Multi-view Pose-based Algorithm for Real-time Human Action Recognition
• Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition
• Learning and Recognizing Human Action from Skeleton Movement with Deep Residual Neural Networks
• Learning Human Motion Models for Long-term Predictions
• First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
• Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition
• Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
• Depth Pooling Based Large-scale 3D Action Recognition with Convolutional Neural Networks
• Action Recognition with Joint Attention on Multi-Level Deep Features
• Concurrence-Aware Long Short-Term Sub-Memories for Person-Person Action Recognition
• Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates
• Learning and Refining of Privileged Information-based RNNs for Action Recognition from Depth Sequences
• Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition
• Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks
• Robust 3D Action Recognition through Sampling Local Appearances and Global Distributions
• Transition Forests: Learning Discriminative Temporal Transitions for Action Recognition and Detection
• RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
• Summarization of ICU Patient Motion from Multimodal Multiview Videos
• Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module
• Discriminatively Learned Hierarchical Rank Pooling Networks
• STS Classification with Dual-stream CNN
• DeepActsNet: Spatial and Motion features from Face, Hands, and Body Combined with Convolutional and Graph Networks for Improved Action Recognition
• Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks