Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation

Abstract

We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

arXiv commentNeurIPS 2021. Project page: https://zhengyiluo.github.io/projects/kin_poly/

References in corpus (6)

• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
• Character Controllers Using Motion VAEs
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• Hierarchical Kinematic Human Mesh Recovery

Cited by in corpus (2)

• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors