DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
arXiv:2308.02915 · doi:10.1145/3581783.3612307
Abstract
When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. To bridge the gap between music and motion for conditional generation, DiffDance employs a pretrained audio representation learning model to extract music embeddings and further align its embedding space to motion via contrastive loss. During training our cascaded diffusion model, we also incorporate multiple geometric losses to constrain the model outputs to be physically plausible and add a dynamic loss weight that adaptively changes over diffusion timesteps to facilitate sample diversity. Through comprehensive experiments performed on the benchmark dataset AIST++, we demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music. These results are comparable to those achieved by state-of-the-art autoregressive methods.
arXiv commentAccepted at ACM MM 2023
References in corpus (25)
- Decoupled Weight Decay Regularization
- Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Generative Modeling by Estimating Gradients of the Data Distribution
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- Classifier-Free Diffusion Guidance
- Cascaded Diffusion Models for High Fidelity Image Generation
- Imagen Video: High Definition Video Generation with Diffusion Models
- Diffusion-LM Improves Controllable Text Generation
- Self-Supervised MultiModal Versatile Networks
- LION: Latent Point Diffusion Models for 3D Shape Generation
- Human Motion Diffusion Model
- Audio to Body Dynamics
- MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
- MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency
- Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio
- Learning to Generate Diverse Dance Motions with Transformer
- Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
- Deep representation learning for human motion prediction and classification
- Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network
- FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
- On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
- Music2Dance: DanceNet for Music-driven Dance Generation