Human Action Recognition from Various Data Modalities: A Review
arXiv:2012.11866 · doi:10.1109/TPAMI.2022.3183112
Abstract
Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.
References in corpus (88)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- A Survey on Visual Transformer
- Transformers in Vision: A Survey
- T-GCN: A Temporal Graph ConvolutionalNetwork for Traffic Prediction
- The Kinetics Human Action Video Dataset
- Squeeze-and-Excitation Networks
- Longformer: The Long-Document Transformer
- NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
- Unsupervised Learning of Video Representations using LSTMs
- Is Space-Time Attention All You Need for Video Understanding?
- Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- YouTube-8M: A Large-Scale Video Classification Benchmark
- Generalizing from a Few Examples: A Survey on Few-Shot Learning
- A New Representation of Skeleton Sequences for 3D Action Recognition
- Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks
- Spatiotemporal Residual Networks for Video Action Recognition
- Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
- Towards Good Practices for Very Deep Two-Stream ConvNets
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
- Rank Pooling for Action Recognition
- A Short Note about Kinetics-600
- Skeleton-based Action Recognition with Convolutional Neural Networks
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Spatial Temporal Transformer Network for Skeleton-based Action Recognition
- Joint Activity Recognition and Indoor Localization with WiFi Fingerprints
- SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
- Attentional Pooling for Action Recognition
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
- Graph-based Spatial-temporal Feature Learning for Neuromorphic Vision Sensing
- Audiovisual SlowFast Networks for Video Recognition
- Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
- PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
- View-invariant Deep Architecture for Human Action Recognition using late fusion
- Long-term Temporal Convolutions for Action Recognition
- UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
- Two-Stream RNN/CNN for Action Recognition in 3D Videos
- Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition
- RegionViT: Regional-to-Local Attention for Vision Transformers
- Ensemble One-dimensional Convolution Neural Networks for Skeleton-based Action Recognition
- Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos
- A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
- Learning with privileged information via adversarial discriminative modality distillation
- Synthetic Humans for Action Recognition from Unseen Viewpoints
- Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
- PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
- IA-RED: Interpretability-Aware Redundancy Reduction for Vision Transformers
- Infrared and 3D skeleton feature fusion for RGB-D action recognition
- Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation
- Space-time Mixing Attention for Video Transformer
- Skeleton-based Relational Reasoning for Group Activity Analysis
- V4D:4D Convolutional Neural Networks for Video-level Representation Learning
- Two-Stream Action Recognition-Oriented Video Super-Resolution
- Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition
- Deep Independently Recurrent Neural Network (IndRNN)
- Dynamic Vision Sensors for Human Activity Recognition
- Motion and Region Aware Adversarial Learning for Fall Detection with Thermal Imaging
- Spatiotemporal Filtering for Event-Based Action Recognition
- CT-Net: Channel Tensorization Network for Video Classification
- Temporal Pyramid Network for Action Recognition
- Anchor-Based Spatio-Temporal Attention 3D Convolutional Networks for Dynamic 3D Point Cloud Sequences
- Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
- Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
- Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition
- STAR: Sparse Transformer-based Action Recognition
- Temporal Unet: Sample Level Human Action Recognition using WiFi
- RareAct: A video dataset of unusual interactions
- DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition
- Model-agnostic Multi-Domain Learning with Domain-Specific Adapters for Action Recognition
- Can An Image Classifier Suffice For Action Recognition?
- 3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information
- MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
- Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment
- Simultaneous Energy Harvesting and Gait Recognition using Piezoelectric Energy Harvester
- Few-Shot Action Localization without Knowing Boundaries
- Event-based Action Recognition Using Timestamp Image Encoding Network
- E(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition
- Selective Feature Compression for Efficient Activity Recognition Inference
Cited by in corpus (11)
- From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
- D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation
- Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition
- Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
- Human I/O: Towards a Unified Approach to Detecting Situational Impairments
- EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition
- A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
- TRTAR: Transmissive RIS-assisted Through-the-wall Human Activity Recognition
- Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition
- Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives
- Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification