Human Action Recognition from Various Data Modalities: A Review

Abstract

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

References in corpus (51)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• The Kinetics Human Action Video Dataset
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Is Space-Time Attention All You Need for Video Understanding?
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Spatiotemporal Residual Networks for Video Action Recognition
• Towards Good Practices for Very Deep Two-Stream ConvNets
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Spatial Temporal Transformer Network for Skeleton-based Action Recognition
• Joint Activity Recognition and Indoor Localization with WiFi Fingerprints
• Attentional Pooling for Action Recognition
• Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• View-invariant Deep Architecture for Human Action Recognition using late fusion
• UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition
• RegionViT: Regional-to-Local Attention for Vision Transformers
• PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Space-time Mixing Attention for Video Transformer
• V4D:4D Convolutional Neural Networks for Video-level Representation Learning
• Skeleton-based Relational Reasoning for Group Activity Analysis
• Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition
• Spatiotemporal Filtering for Event-Based Action Recognition
• CT-Net: Channel Tensorization Network for Video Classification
• Anchor-Based Spatio-Temporal Attention 3D Convolutional Networks for Dynamic 3D Point Cloud Sequences
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
• Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition
• Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
• Temporal Unet: Sample Level Human Action Recognition using WiFi
• STAR: Sparse Transformer-based Action Recognition
• RareAct: A video dataset of unusual interactions
• DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition
• Model-agnostic Multi-Domain Learning with Domain-Specific Adapters for Action Recognition
• Can An Image Classifier Suffice For Action Recognition?
• MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
• 3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information
• Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment
• Few-Shot Action Localization without Knowing Boundaries
• Simultaneous Energy Harvesting and Gait Recognition using Piezoelectric Energy Harvester
• Event-based Action Recognition Using Timestamp Image Encoding Network
• E${}^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition
• Selective Feature Compression for Efficient Activity Recognition Inference

Cited by in corpus (11)

• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition
• Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
• Human I/O: Towards a Unified Approach to Detecting Situational Impairments
• EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition
• A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
• TRTAR: Transmissive RIS-assisted Through-the-wall Human Activity Recognition
• Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition
• Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives
• Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification