Learning a Latent Space of Multitrack Measures

Abstract

Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem. We extend the recent MusicVAE model to represent multitrack polyphonic measures as vectors in a latent space. Our approach enables several useful operations such as generating plausible measures from scratch, interpolating between measures in a musically meaningful way, and manipulating specific musical attributes. We also introduce chord conditioning, which allows all of these operations to be performed while keeping harmony fixed, and allows chords to be changed while maintaining musical "style". By generating a sequence of measures over a predefined chord progression, our model can produce music with convincing long-term structure. We demonstrate that our latent space model makes it possible to intuitively control and generate musical sequences with rich instrumentation (see https://goo.gl/s2N7dV for generated audio).

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• A Neural Representation of Sketch Drawings
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
• DeepBach: a Steerable Model for Bach Chorales Generation
• Counterpoint by Convolution
• Sampling Variations of Lead Sheets

Cited by in corpus (14)

• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Learning to Traverse Latent Spaces for Musical Score Inpainting
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Learning Disentangled Representations for Timber and Pitch in Music Audio
• Symbolic Music Generation with Diffusion Models
• SurpriseNet: Melody Harmonization Conditioning on User-controlled Surprise Contours
• Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Modeling Melodic Feature Dependency with Modularized Variational Auto-Encoder
• Symbolic Music Loop Generation with VQ-VAE
• Using Shapley Values and Variational Autoencoders to Explain Predictive Models with Dependent Mixed Features
• PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network
• Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation