Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery

Abstract

Regression-based methods have shown high efficiency and effectiveness for multi-view human mesh recovery. The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features. In this paper, we present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. At each iteration, PaFF extracts pixel-aligned feedback features from each input view according to the reprojection of the current estimation and fuses them together with respect to each vertex of the downsampled mesh. In this way, our regressor can not only perceive the misalignment status of each view from the feedback features but also correct the mesh parameters more effectively based on the feature fusion on mesh vertices. Additionally, our regressor disentangles the global orientation and translation of the body mesh from the estimation of mesh parameters such that the camera parameters of input views can be better utilized in the regression process. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments, where PaFF achieves 33.02 MPJPE and brings significant improvements over the previous best solutions by more than 29%. The project page with code and video results can be found at https://kairobo.github.io/PaFF/.

arXiv commentProject Page: https://kairobo.github.io/PaFF/

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Deformable ConvNets v2: More Deformable, Better Results
• Recovering 3D Human Mesh from Monocular Images: A Survey
• PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
• Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation
• Direct Multi-view Multi-person 3D Pose Estimation
• Towards Accurate Markerless Human Shape and Pose Estimation over Time
• On the Continuity of Rotation Representations in Neural Networks
• Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild
• Multi-view Human Pose and Shape Estimation Using Learnable Volumetric Aggregation
• Interacting Attention Graph for Single Image Two-Hand Reconstruction
• Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images
• VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment
• PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
• Learning 3D Human Shape and Pose from Dense Body Parts
• PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction
• Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
• Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images
• 4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras
• Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild
• Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras