Lets Play Music: Audio-driven Performance Video Generation

Abstract

We propose a new task named Audio-driven Per-formance Video Generation (APVG), which aims to synthesizethe video of a person playing a certain instrument guided bya given music audio clip. It is a challenging task to gener-ate the high-dimensional temporal consistent videos from low-dimensional audio modality. In this paper, we propose a multi-staged framework to achieve this new task to generate realisticand synchronized performance video from given music. Firstly,we provide both global appearance and local spatial informationby generating the coarse videos and keypoints of body and handsfrom a given music respectively. Then, we propose to transformthe generated keypoints to heatmap via a differentiable spacetransformer, since the heatmap offers more spatial informationbut is harder to generate directly from audio. Finally, wepropose a Structured Temporal UNet (STU) to extract bothintra-frame structured information and inter-frame temporalconsistency. They are obtained via graph-based structure module,and CNN-GRU based high-level temporal module respectively forfinal video generation. Comprehensive experiments validate theeffectiveness of our proposed framework.

arXiv commentICPR 2020

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• ObamaNet: Photo-realistic lip-sync from text
• Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
• Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Speech-Driven Facial Reenactment Using Conditional Generative Adversarial Networks
• Geometry-Aware Face Completion and Editing