Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
arXiv:2106.05266
Abstract
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a single image perfectly. To tackle these challenges, we propose a unified framework for estimating the 3D hand and object poses with semi-supervised learning. We build a joint learning framework where we perform explicit contextual reasoning between hand and object representations by a Transformer. Going beyond limited 3D annotations in a single image, we leverage the spatial-temporal consistency in large-scale hand-object videos as a constraint for generating pseudo labels in semi-supervised learning. Our method not only improves hand pose estimation in challenging real-world dataset, but also substantially improve the object pose which has fewer ground-truths per instance. By training with large-scale diverse videos, our model also generalizes better across multiple out-of-domain datasets. Project page and code: https://stevenlsw.github.io/Semi-Hand-Object
CVPR 2021, Project page: https://stevenlsw.github.io/Semi-Hand-Object
References in corpus (13)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- LSTM: A Search Space Odyssey
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Interaction Networks for Learning about Objects, Relations and Physics
- Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
- Self-training with Noisy Student improves ImageNet classification
- Unifying distillation and privileged information
- Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
- TFPose: Direct Human Pose Estimation with Transformers
- Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
- End-to-End Trainable Multi-Instance Pose Estimation with Transformers
- Revisiting the Continuity of Rotation Representations in Neural Networks