MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer

Abstract

We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce medleys and create mixtures of entire songs. The interpolations smoothly change pitches, dynamics and instrumentation to create a harmonic bridge between two music pieces. To the best of our knowledge, this work represents the first successful attempt at applying neural style transfer to complete musical compositions.

arXiv commentPaper accepted at the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• WaveNet: A Generative Model for Raw Audio
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• C-RNN-GAN: Continuous recurrent neural networks with adversarial training
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Neural Translation of Musical Style

Cited by in corpus (9)

• Deep Music Analogy Via Latent Representation Disentanglement
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Learning Disentangled Representations for Timber and Pitch in Music Audio
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• MIDI-Sandwich2: RNN-based Hierarchical Multi-modal Fusion Generation VAE networks for multi-track symbolic music generation
• Inspecting and Interacting with Meaningful Music Representations using VAE
• Musical Composition Style Transfer via Disentangled Timbre Representations
• Neural Melody Composition from Lyrics
• Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network