MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency

Abstract

We introduce MotioNet, a deep neural network that directly reconstructs the motion of a 3D human skeleton from monocular video.While previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used, motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes: a single, symmetric, skeleton, encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels. These attributes are fed into an integrated forward kinematics (FK) layer that outputs 3D positions, which are compared to a ground truth. In addition, an adversarial loss is applied to the velocities of the recovered rotations, to ensure that they lie on the manifold of natural joint rotations. The key advantage of our approach is that it learns to infer natural joint rotations directly from the training data, rather than assuming an underlying model, or inferring them from joint positions using a data-agnostic IK solver. We show that enforcing a single consistent skeleton along with temporally coherent joint rotations constrains the solution space, leading to a more robust handling of self-occlusions and depth ambiguities.

arXiv commentAccepted to Transactions on Graphics (ToG) 2020. Project page: {https://rubbly.cn/publications/motioNet} Video: {https://youtu.be/8YubchlzvFA}

References in corpus (5)

• DeepPose: Human Pose Estimation via Deep Neural Networks
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• QuaterNet: A Quaternion-based Recurrent Model for Human Motion
• DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Cited by in corpus (13)

• Recovering 3D Human Mesh from Monocular Images: A Survey
• Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation
• A Survey on Deep Learning for Skeleton-Based Human Animation
• World-Grounded Human Motion Recovery via Gravity-View Coordinates
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
• Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality
• Pose Representations for Deep Skeletal Animation
• A Two-part Transformer Network for Controllable Motion Synthesis
• Conditional Directed Graph Convolution for 3D Human Pose Estimation
• Task-Generic Hierarchical Human Motion Prior using VAEs
• Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques
• Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs