Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation

Abstract

We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation using a head-mounted camera. We use a kinematics model of the human body to represent the entire range of human motion, and a dynamics model of the body to interact with objects inside a physics simulator. By bringing together object modeling, kinematics modeling, and dynamics modeling in a reinforcement learning (RL) framework, we enable object-aware 3D ego-pose estimation. We devise several representational innovations through the design of the state and action space to incorporate 3D scene context and improve pose estimation quality. We also construct a fine-tuning step to correct the drift and refine the estimated human-object interaction. This is the first work to estimate a physically valid 3D full-body interaction sequence with objects (e.g., chairs, boxes, obstacles) from egocentric videos. Experiments with both controlled and in-the-wild settings show that our method can successfully extract an object-conditioned 3D ego-pose sequence that is consistent with the laws of physics.

arXiv commentProject website: https://zhengyiluo.github.io/projects/contextegopose/

References in corpus (4)

• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control