GRAB: A Dataset of Whole-Body Human Grasping of Objects

Abstract

Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at https://grab.is.tue.mpg.de.

ECCV 2020

References in corpus (3)

• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video
• Learning Multi-Human Optical Flow

Cited by in corpus (18)

• Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator
• Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions
• UGG: Unified Generative Grasping
• Synthesize Dexterous Nonprehensile Pregrasp for Ungraspable Objects
• SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers
• POV-Surgery: A Dataset for Egocentric Hand and Tool Pose Estimation During Surgical Activities
• DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions
• A Grasp Pose is All You Need: Learning Multi-fingered Grasping with Deep Reinforcement Learning from Vision and Touch
• Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
• Shaping the Future of VR Hand Interactions: Lessons Learned from Modern Methods
• A Survey on Human Interaction Motion Generation
• DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations
• Personalized 3D Human Pose and Shape Refinement
• RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture
• Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration
• BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
• Manipulate as Human: Learning Task-oriented Manipulation Skills by Adversarial Motion Priors
• ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulation