Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network

Abstract

Automatic choreography generation is a challenging task because it often requires an understanding of two abstract concepts - music and dance - which are realized in the two different modalities, namely audio and video, respectively. In this paper, we propose a music-driven choreography generation system using an auto-regressive encoder-decoder network. To this end, we first collect a set of multimedia clips that include both music and corresponding dance motion. We then extract the joint coordinates of the dancer from video and the mel-spectrogram of music from audio, and train our network using music-choreography pairs as input. Finally, a novel dance motion is generated at the inference time when only music is given as an input. We performed a user study for a qualitative evaluation of the proposed method, and the results show that the proposed model is able to generate musically meaningful and natural dance movements given an unheard song.

arXiv comment5 pages

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• WaveNet: A Generative Model for Raw Audio
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention

Cited by in corpus (7)

• Learning to Generate Diverse Dance Motions with Transformer
• Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• Music-oriented Dance Video Synthesis with Pose Perceptual Loss
• Feel The Music: Automatically Generating A Dance For An Input Song
• Lets Play Music: Audio-driven Performance Video Generation