DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation

Abstract

When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. To bridge the gap between music and motion for conditional generation, DiffDance employs a pretrained audio representation learning model to extract music embeddings and further align its embedding space to motion via contrastive loss. During training our cascaded diffusion model, we also incorporate multiple geometric losses to constrain the model outputs to be physically plausible and add a dynamic loss weight that adaptively changes over diffusion timesteps to facilitate sample diversity. Through comprehensive experiments performed on the benchmark dataset AIST++, we demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music. These results are comparable to those achieved by state-of-the-art autoregressive methods.

Accepted at ACM MM 2023

References in corpus (11)

• Diffusion Models Beat GANs on Image Synthesis
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Classifier-Free Diffusion Guidance
• Imagen Video: High Definition Video Generation with Diffusion Models
• Diffusion-LM Improves Controllable Text Generation
• LION: Latent Point Diffusion Models for 3D Shape Generation
• Human Motion Diffusion Model
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
• Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio
• Learning to Generate Diverse Dance Motions with Transformer
• FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis