Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion

Abstract

Accurate 3D reconstruction of the hand and object shape from a hand-object image is important for understanding human-object interaction as well as human daily activities. Different from bare hand pose estimation, hand-object interaction poses a strong constraint on both the hand and its manipulated object, which suggests that hand configuration may be crucial contextual information for the object, and vice versa. However, current approaches address this task by training a two-branch network to reconstruct the hand and object separately with little communication between the two branches. In this work, we propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We extensively investigate cross-branch feature fusion architectures with MLP or LSTM units. Among the investigated architectures, a variant with LSTM units that enhances object feature with hand feature shows the best performance gain. Moreover, we employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map, which further improves the reconstruction accuracy. Experiments conducted on public datasets demonstrate that our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.

arXiv commentAccepted by IEEE Transactions on Image Processing (TIP)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• ShapeNet: An Information-Rich 3D Model Repository
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Region Ensemble Network: Improving Convolutional Network for Hand Pose Estimation
• Learning to Reconstruct Shapes from Unseen Classes
• Learning to Infer and Execute 3D Shape Programs
• 3D Object Reconstruction from Hand-Object Interactions
• GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects

Cited by in corpus (8)

• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation
• Consistent 3D Hand Reconstruction in Video via self-supervised Learning
• Deep learning for 3D human pose estimation and mesh recovery: A survey
• Model-based 3D Hand Reconstruction via Self-Supervised Learning
• EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
• PEAR: Phrase-Based Hand-Object Interaction Anticipation
• Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
• Sequential Voting with Relational Box Fields for Active Object Detection