PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images

Abstract

We present PyMAF-X, a regression-based approach to recovering parametric full-body models from monocular images. This task is very challenging since minor parametric deviation may lead to noticeable misalignment between the estimated mesh and the input image. Moreover, when integrating part-specific estimations into the full-body model, existing solutions tend to either degrade the alignment or produce unnatural wrist poses. To address these issues, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop in our regression network for well-aligned human mesh recovery and extend it as PyMAF-X for the recovery of expressive full-body models. The core idea of PyMAF is to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status. Specifically, given the currently predicted parameters, mesh-aligned evidence will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To enhance the alignment perception, an auxiliary dense supervision is employed to provide mesh-image correspondence guidance while spatial alignment attention is introduced to enable the awareness of the global contexts for our network. When extending PyMAF for full-body mesh recovery, an adaptive integration strategy is proposed in PyMAF-X to produce natural wrist poses while maintaining the well-aligned performance of the part-specific estimations. The efficacy of our approach is validated on several benchmark datasets for body, hand, face, and full-body mesh recovery, where PyMAF and PyMAF-X effectively improve the mesh-image alignment and achieve new state-of-the-art results. The project page with code and video results can be found at https://zhanghongwen.cn/pymaf-x.

Article in IEEE TPAMI 2023, Update project page: https://zhanghongwen.cn/pymaf-x, An eXpressive extension of PyMAF [arXiv:2103.16507] for monocular human/hand/face/whole-body motion capture

References in corpus (4)

• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Recovering 3D Human Mesh from Monocular Images: A Survey
• DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image
• Swing-twist decomposition in Clifford algebra

Cited by in corpus (8)

• Recovering 3D Human Mesh from Monocular Images: A Survey
• World-Grounded Human Motion Recovery via Gravity-View Coordinates
• Deep learning for 3D human pose estimation and mesh recovery: A survey
• Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery
• KBody: Towards general, robust, and aligned monocular whole-body estimation
• HGC-Avatar: Hierarchical Gaussian Compression for Streamable Dynamic 3D Avatars
• FAMOUS: High-Fidelity Monocular 3D Human Digitization Using View Synthesis
• ClothHMR: 3D Mesh Recovery of Humans in Diverse Clothing from Single Image