Embodied Hands: Modeling and Capturing Hands and Bodies Together
arXiv:2201.02610 · doi:10.1145/3130800.3130883
Abstract
Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid defOrmations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes in our website (http://mano.is.tue.mpg.de).
arXiv commentSIGGRAPH ASIA 2017
References in corpus (4)
Cited by in corpus (148)
- GRAB: A Dataset of Whole-Body Human Grasping of Objects
- Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
- Recovering 3D Human Mesh from Monocular Images: A Survey
- PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
- SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding
- Differentiable Rendering: A Survey
- Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator
- An Implicit Parametric Morphable Dental Model
- Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
- Consistent 3D Hand Reconstruction in Video via self-supervised Learning
- UmeTrack: Unified multi-view end-to-end hand tracking for VR
- Orienting Point Clouds with Dipole Propagation
- Modeling Clothing as a Separate Layer for an Animatable Human Avatar
- 3D Hand Pose and Shape Estimation from RGB Images for Keypoint-Based Hand Gesture Recognition
- FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
- The Metaverse: Survey, Trends, Novel Pipeline Ecosystem & Future Directions
- Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
- 3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View
- Disentangled Human Body Embedding Based on Deep Hierarchical Neural Network
- ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wild
- End-to-End Human Pose and Mesh Reconstruction with Transformers
- Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
- Force-Aware Interface via Electromyography for Natural VR/AR Interaction
- 3D Hand Shape and Pose Estimation from a Single RGB Image
- Efficient Neural Implicit Representation for 3D Human Reconstruction
- History Repeats Itself: Human Motion Prediction via Motion Attention
- StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition
- Monocular Total Capture: Posing Face, Body, and Hands in the Wild
- Detecting Hands and Recognizing Physical Contact in the Wild
- UGG: Unified Generative Grasping
- DeepCap: Monocular Human Performance Capture Using Weak Supervision
- HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
- Look Ma, no markers: holistic performance capture without the hassle
- Single Image 3D Hand Reconstruction with Mesh Convolutions
- SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers
- GraFormer: Graph Convolution Transformer for 3D Pose Estimation
- HandAugment: A Simple Data Augmentation Method for Depth-Based 3D Hand Pose Estimation
- HandTailor: Towards High-Precision Monocular 3D Hand Recovery
- End-to-end Hand Mesh Recovery from a Monocular RGB Image
- LaplacianFusion: Detailed 3D Clothed-Human Body Reconstruction
- Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
- Neural-ABC: Neural Parametric Models for Articulated Body with Clothes
- Grasping Field: Learning Implicit Representations for Human Grasps
- The Whole Is Greater Than the Sum of Its Nonrigid Parts
- Neural Radiance Flow for 4D View Synthesis and Video Processing
- DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions
- Neural Rendering and Reenactment of Human Actor Videos
- DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
- Recent Advances in 3D Object and Hand Pose Estimation
- Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
- SimulCap : Single-View Human Performance Capture with Cloth Simulation
- A Locality-based Neural Solver for Optical Motion Capture
- LiveCap: Real-time Human Performance Capture from Monocular Video
- 4D Human Body Capture from Egocentric Video via 3D Scene Grounding
- Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
- Model-based 3D Hand Reconstruction via Self-Supervised Learning
- Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images
- 3D Visual Tracking to Quantify Physical Contact Interactions in Human-to-Human Touch
- Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge
- Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
- Understanding Human Hands in Contact at Internet Scale
- We are More than Our Joints: Predicting how 3D Bodies Move
- DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
- PoseScript: Linking 3D Human Poses and Natural Language
- A Modular Bio-inspired Robotic Hand with High Sensitivity
- Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
- Hand-Object Contact Consistency Reasoning for Human Grasps Generation
- Monocular Real-time Full Body Capture with Inter-part Correlations
- Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
- HUMBI: A Large Multiview Dataset of Human Body Expressions
- GraV: Grasp Volume Data for the Design of One-Handed XR Interfaces
- Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction
- Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses
- SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks
- CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction
- Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration
- Semantics2Hands: Transferring Hand Motion Semantics between Avatars
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
- A Survey on Human Interaction Motion Generation
- Point-to-Pose Voting based Hand Pose Estimation using Residual Permutation Equivariant Layer
- SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation
- Two-hand Global 3D Pose Estimation Using Monocular RGB
- SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
- Whole-Body Human Pose Estimation in the Wild
- Monocular Real-Time Volumetric Performance Capture
- PEAR: Phrase-Based Hand-Object Interaction Anticipation
- ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis
- Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction
- Graph Based Gaussian Processes on Restricted Domains
- Creating a Large-scale Synthetic Dataset for Human Activity Recognition
- Unsupervised Shape and Pose Disentanglement for 3D Meshes
- Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
- S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling
- Unsupervised Domain Adaptation with Temporal-Consistent Self-Training for 3D Hand-Object Joint Reconstruction
- Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos
- Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
- Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras
- HaDR: Applying Domain Randomization for Generating Synthetic Multimodal Dataset for Hand Instance Segmentation in Cluttered Industrial Environments
- You Can Wash Hands Better: Accurate Daily Handwashing Assessment with a Smartwatch
- Motion Capture from Internet Videos
- DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations
- HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images
- A Modular Approach to the Embodiment of Hand Motions from Human Demonstrations
- Learning Feature Aggregation for Deep 3D Morphable Models
- SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
- Learning Localization of Body and Finger Animation Skeleton Joints on Three-Dimensional Models of Human Bodies
- Identity-Disentangled Neural Deformation Model for Dynamic Meshes
- MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
- Dependent landmark drift: robust point set registration with a Gaussian mixture model and a statistical shape model
- Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
- HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset
- Reconstructing NBA Players
- Stereo Hand-Object Reconstruction for Human-to-Robot Handover
- Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies
- DC-GNet: Deep Mesh Relation Capturing Graph Convolution Network for 3D Human Shape Reconstruction
- Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation
- Physically Plausible Pose Refinement using Fully Differentiable Forces
- GraspR: A Computational Model of Spatial User Preferences for Adaptive Grasp UI Design
- Synthetic Training for Monocular Human Mesh Recovery
- 3D Hand Pose Estimation via Regularized Graph Representation Learning
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
- A novel shape matching descriptor for real-time hand gesture recognition
- Learning Transferable Kinematic Dictionary for 3D Human Pose and Shape Reconstruction
- Parallel mesh reconstruction streams for pose estimation of interacting hands
- PCHands: PCA-based Hand Pose Synergy Representation on Manipulators with N-DoF
- LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
- Multi-Resolution Generative Modeling of Human Motion from Limited Data
- Hierarchical Neural Implicit Pose Network for Animation and Motion Retargeting
- Self-supervised Optimization of Hand Pose Estimation using Anatomical Features and Iterative Learning
- 3DHR-Co: A Collaborative Test-time Refinement Framework for In-the-Wild 3D Human-Body Reconstruction Task
- LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies
- Rotation Equivariant 3D Hand Mesh Generation from a Single RGB Image
- A Deeper Look into DeepCap
- Hierarchical Graph-Convolutional Variational AutoEncoding for Generative Modelling of Human Motion
- RealisticHands: A Hybrid Model for 3D Hand Reconstruction
- TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes
- Robust Photo-Realistic Hand Gesture Generation: from Single View to Multiple View
- EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream
- EgoPressDiff: Multimodal Video Diffusion for Egocentric UV-Domain Hand-Pressure Estimation
- Birds of a Feather: Capturing Avian Shape Models from Images
- Single RGB-D Camera Teleoperation for General Robotic Manipulation
- Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type
- Deep 3D Mesh Watermarking with Self-Adaptive Robustness
- Learning to Train with Synthetic Humans
- Real-time RGBD-based Extended Body Pose Estimation
- From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding
- Towards Markerless Grasp Capture
- TexturePose: Supervising Human Mesh Estimation with Texture Consistency