Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective

Abstract

Estimating 3D full-body pose from sparse sensor data is a pivotal technique employed for the reconstruction of realistic human motions in Augmented Reality and Virtual Reality. However, translating sparse sensor signals into comprehensive human motion remains a challenge since the sparsely distributed sensors in common VR systems fail to capture the motion of full human body. In this paper, we use well-designed Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes. Then, we propose a novel full-body motion reconstruction framework based on BPG. To establish BPG, nodes are initially endowed with features extracted from sparse sensor signals. Features from identifiable joint nodes across diverse sensors are amalgamated and processed from both temporal and spatial perspectives. Temporal dynamics are captured using the Temporal Pyramid Structure, while spatial relations in joint movements inform the spatial attributes. The resultant features serve as the foundational elements of the BPG nodes. To further refine the BPG, node features are updated through a graph neural network that incorporates edge reflecting varying joint relations. Our method's effectiveness is evidenced by the attained state-of-the-art performance, particularly in lower body motion, outperforming other baseline methods. Additionally, an ablation study validates the efficacy of each module in our proposed framework.

References in corpus (18)

• Semi-Supervised Classification with Graph Convolutional Networks
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
• Graph Convolutional Networks for Graphs Containing Missing Features
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition
• TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors
• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition
• Incomplete Graph Representation and Learning via Partial Graph Neural Networks
• On the Unreasonable Effectiveness of Feature propagation in Learning on Graphs with Missing Node Features
• Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors
• AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing
• Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation
• Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model