Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery

Abstract

Regression-based methods have shown high efficiency and effectiveness for multi-view human mesh recovery. The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features. In this paper, we present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. At each iteration, PaFF extracts pixel-aligned feedback features from each input view according to the reprojection of the current estimation and fuses them together with respect to each vertex of the downsampled mesh. In this way, our regressor can not only perceive the misalignment status of each view from the feedback features but also correct the mesh parameters more effectively based on the feature fusion on mesh vertices. Additionally, our regressor disentangles the global orientation and translation of the body mesh from the estimation of mesh parameters such that the camera parameters of input views can be better utilized in the regression process. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments, where PaFF achieves 33.02 MPJPE and brings significant improvements over the previous best solutions by more than 29%. The project page with code and video results can be found at https://kairobo.github.io/PaFF/.

Project Page: https://kairobo.github.io/PaFF/

References in corpus (11)

• Recovering 3D Human Mesh from Monocular Images: A Survey
• PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
• Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation
• Direct Multi-view Multi-person 3D Pose Estimation
• Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras
• Interacting Attention Graph for Single Image Two-Hand Reconstruction
• Multi-view Human Pose and Shape Estimation Using Learnable Volumetric Aggregation
• PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
• 4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras
• Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images
• Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras