QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars

Abstract

Real-time tracking of human body motion is crucial for interactive and immersive experiences in AR/VR. However, very limited sensor data about the body is available from standalone wearable devices such as HMDs (Head Mounted Devices) or AR glasses. In this work, we present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers, and simulates plausible and physically valid full body motions. Using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balance, walk, and jog, while closely following the input signals. Our results demonstrate surprisingly similar leg motions to ground truth without any observations of the lower body, even when the input is only the 6D transformations of the HMD. We also show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.

References in corpus (14)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Character Controllers Using Motion VAEs
• Robust Motion In-betweening
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
• Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation
• MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies
• TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors
• SFV: Reinforcement Learning of Physical Skills from Videos
• Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
• PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time
• AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing

Cited by in corpus (14)

• LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors
• SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data
• SoK: Data Privacy in Virtual Reality
• HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing
• QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors
• Animation Fidelity in Self-Avatars: Impact on User Performance and Sense of Agency
• Investigating Creation Perspectives and Icon Placement Preferences for On-Body Menus in Virtual Reality
• MOVIN: Real-time Motion Capture using a Single LiDAR
• Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions
• SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation
• Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective
• DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization
• ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling
• OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching