DeepFuse: An IMU-Aware Network for Real-Time 3D Human Pose Estimation from Multi-View Image

Abstract

In this paper, we propose a two-stage fully 3D network, namely \textbf{DeepFuse}, to estimate human pose in 3D space by fusing body-worn Inertial Measurement Unit (IMU) data and multi-view images deeply. The first stage is designed for pure vision estimation. To preserve data primitiveness of multi-view inputs, the vision stage uses multi-channel volume as data representation and 3D soft-argmax as activation layer. The second one is the IMU refinement stage which introduces an IMU-bone layer to fuse the IMU and vision data earlier at data level. without requiring a given skeleton model a priori, we can achieve a mean joint error of $28.9$mm on TotalCapture dataset and $13.4$mm on Human3.6M dataset under protocol 1, improving the SOTA result by a large margin. Finally, we discuss the effectiveness of a fully 3D network for 3D pose estimation experimentally which may benefit future research.

References in corpus (9)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Human pose estimation via Convolutional Part Heatmap Regression
• Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
• Human Pose Regression by Combining Indirect Part Detection and Contextual Information
• Numerical Coordinate Regression with Convolutional Neural Networks
• Synthetic Occlusion Augmentation with Volumetric Heatmaps for the 2018 ECCV PoseTrack Challenge on 3D Human Pose Estimation
• Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs
• 3D Human Pose Estimation with 2D Marginal Heatmaps

Cited by in corpus (1)

• SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach