Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Abstract

Human pose estimation and action recognition are related tasks since both problems are strongly dependent on the human body representation and analysis. Nonetheless, most recent methods in the literature handle the two problems separately. In this work, we propose a multi-task framework for jointly estimating 2D or 3D human poses from monocular color images and classifying human actions from video sequences. We show that a single architecture can be used to solve both problems in an efficient way and still achieves state-of-the-art or comparable results at each task while running at more than 100 frames per second. The proposed method benefits from high parameters sharing between the two tasks by unifying still images and video clips processing in a single pipeline, allowing the model to be trained with data from different categories simultaneously and in a seamlessly way. Additionally, we provide important insights for end-to-end training the proposed multi-task model by decoupling key prediction parts, which consistently leads to better accuracy on both tasks. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU RGB+D) demonstrate the effectiveness of our method on the targeted tasks. Our source code and trained weights are publicly available at https://github.com/dluvizon/deephar.

Accepted to TPAMI. arXiv admin note: text overlap with arXiv:1802.09232

References in corpus (9)

• DeepPose: Human Pose Estimation via Deep Neural Networks
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• A New Representation of Skeleton Sequences for 3D Action Recognition
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Human pose estimation via Convolutional Part Heatmap Regression
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Human Pose Regression by Combining Indirect Part Detection and Contextual Information
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
• Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition

Cited by in corpus (18)

• Human Action Recognition from Various Data Modalities: A Review
• Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition
• Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
• MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
• Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation
• The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation
• Deep learning for 3D human pose estimation and mesh recovery: A survey
• Dual networks based 3D Multi-Person Pose Estimation from Monocular Video
• UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition
• View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose
• Towards High Performance One-Stage Human Pose Estimation
• View-Invariant Probabilistic Embedding for Human Pose
• A Proposed Set of Communicative Gestures for Human Robot Interaction and an RGB Image-based Gesture Recognizer Implemented in ROS
• EAGLE: Egocentric AGgregated Language-video Engine
• Can Super Resolution be used to improve Human Pose Estimation in Low Resolution Scenarios?
• Selective Spatio-Temporal Aggregation Based Pose Refinement System: Towards Understanding Human Activities in Real-World Videos
• Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization
• Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images