Concurrent Activity Recognition with Multimodal CNN-LSTM Structure

Abstract

We introduce a system that recognizes concurrent activities from real-world data captured by multiple sensors of different types. The recognition is achieved in two steps. First, we extract spatial and temporal features from the multimodal data. We feed each datatype into a convolutional neural network that extracts spatial features, followed by a long-short term memory network that extracts temporal information in the sensory data. The extracted features are then fused for decision making in the second step. Second, we achieve concurrent activity recognition with a single classifier that encodes a binary output vector in which elements indicate whether the corresponding activity types are currently in progress. We tested our system with three datasets from different domains recorded using different sensors and achieved performance comparable to existing systems designed specifically for those domains. Our system is the first to address the concurrent activity recognition with multisensory data using a single model, which is scalable, simple to train and easy to deploy.

14 pages, 12 figures, under review

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Residual Learning for Image Recognition
• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
• Understanding Neural Networks Through Deep Visualization
• Visualizing and Understanding Recurrent Networks
• Voice Recognition Algorithms using Mel Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) Techniques
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling

Cited by in corpus (7)

• Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
• A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos
• Towards Multi-User Activity Recognition through Facilitated Training Data and Deep Learning for Human-Robot Collaboration Applications
• Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations
• Tri-axial Self-Attention for Concurrent Activity Recognition
• Applying Incremental Deep Neural Networks-based Posture Recognition Model for Injury Risk Assessment in Construction