Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose

Abstract

We present a method for human pose tracking that is based on learning spatiotemporal relationships among joints. Beyond generating the heatmap of a joint in a given frame, our system also learns to predict the offset of the joint from a neighboring joint in the frame. Additionally, it is trained to predict the displacement of the joint from its position in the previous frame, in a manner that can account for possibly changing joint appearance, unlike optical flow. These relational cues in the spatial domain and temporal domain are inferred in a robust manner by attending only to relevant areas in the video frames. By explicitly learning and exploiting these joint relationships, our system achieves state-of-the-art performance on standard benchmarks for various pose tracking tasks including 3D body pose tracking in RGB video, 3D hand pose tracking in depth sequences, and 3D hand gesture tracking in RGB video.

References in corpus (23)

• Sequence to Sequence Learning with Neural Networks
• Recurrent Neural Network Regularization
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Human pose estimation via Convolutional Part Heatmap Regression
• Pose Guided Structured Region Ensemble Network for Cascaded Hand Pose Estimation
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• 3D Hand Pose Tracking and Estimation Using Stereo Matching
• Simple Baselines for Human Pose Estimation and Tracking
• A simple yet effective baseline for 3d human pose estimation
• Model-based Deep Hand Pose Estimation
• Towards Accurate Multi-person Pose Estimation in the Wild
• Structured Prediction of 3D Human Pose with Deep Neural Networks
• Synthetic Occlusion Augmentation with Volumetric Heatmaps for the 2018 ECCV PoseTrack Challenge on 3D Human Pose Estimation
• Joint Flow: Temporal Flow Fields for Multi Person Tracking
• MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose
• 3D Human Pose Estimation from a Single Image via Distance Matrix Regression
• 3D Human Pose Estimation = 2D Pose Estimation + Matching
• PoseTrack: Joint Multi-Person Pose Estimation and Tracking