Deep Music Analogy Via Latent Representation Disentanglement

Abstract

Analogy-making is a key method for computer algorithms to generate both natural and creative music pieces. In general, an analogy is made by partially transferring the music abstractions, i.e., high-level representations and their relationships, from one piece to another; however, this procedure requires disentangling music representations, which usually takes little effort for musicians but is non-trivial for computers. Three sub-problems arise: extracting latent representations from the observation, disentangling the representations so that each part has a unique semantic interpretation, and mapping the latent representations back to actual music. In this paper, we contribute an explicitly-constrained variational autoencoder (EC${}^2$-VAE) as a unified solution to all three sub-problems. We focus on disentangling the pitch and rhythm representations of 8-beat music clips conditioned on chords. In producing music analogies, this model helps us to realize the imaginary situation of "what if" a piece is composed using a different pitch contour, rhythm pattern, or chord progression by borrowing the representations from other pieces. Finally, we validate the proposed disentanglement method using objective measurements and evaluate the analogy examples by a subjective study.

arXiv commentAccepted at the International Society for Music Information Retrieval (ISMIR), 2019

References in corpus (9)

• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Tutorial on Variational Autoencoders
• Disentangling by Factorising
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
• MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer
• Music Style Transfer: A Position Paper
• Inspecting and Interacting with Meaningful Music Representations using VAE

Cited by in corpus (10)

• Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions
• Controllable deep melody generation via hierarchical music structure representation
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends
• AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer
• Personalized Popular Music Generation Using Imitation and Structure
• Word Representation for Rhythms