Foley Music: Learning to Generate Music from Videos

Abstract

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the demo video with audio turned on to experience the results.

arXiv commentECCV 2020. Project page: http://foley-music.csail.mit.edu

References in corpus (10)

• WaveNet: A Generative Model for Raw Audio
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Generating Visually Aligned Sound from Videos
• Self-Supervised Generation of Spatial Audio for 360 Video
• Audeo: Audio Generation for a Silent Performance Video
• Song From PI: A Musically Plausible Network for Pop Music Generation

Cited by in corpus (6)

• Video Background Music Generation with Controllable Music Transformer
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
• Taming Visually Guided Sound Generation
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Dance2Music: Automatic Dance-driven Music Generation
• One-shot Scene Graph Generation