BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics

Abstract

The recently emerging text-to-motion advances have spired numerous attempts for convenient and interactive human motion generation. Yet, existing methods are largely limited to generating body motions only without considering the rich two-hand motions, let alone handling various conditions like body dynamics or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands and provides pair-wised finger-level hand annotations and body descriptions. We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts. We first warm up two parallel body-to-hand and text-to-hand diffusion models and then utilize the cross-attention transformer for motion blending. Extensive experiments and cross-validations demonstrate the effectiveness of our approach and dataset for generating convincing two-hand motions from the hybrid body-and-textual conditions. Our dataset and code will be disseminated to the community for future research.

Accepted to CVPR 2024

References in corpus (14)

• Denoising Diffusion Probabilistic Models
• Diffusion Models Beat GANs on Image Synthesis
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
• The KIT Motion-Language Dataset
• GRAB: A Dataset of Whole-Body Human Grasping of Objects
• Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks
• InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
• Audio-Driven Co-Speech Gesture Video Generation
• Modiff: Action-Conditioned 3D Motion Generation with Denoising Diffusion Probabilistic Models
• NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions
• Generative Proxemics: A Prior for 3D Social Interaction from Images
• HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models