Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
arXiv:2007.15474
Abstract
High-level musical qualities (such as emotion) are often abstract, subjective, and hard to quantify. Given these difficulties, it is not easy to learn good feature representations with supervised learning techniques, either because of the insufficiency of labels, or the subjectiveness (and hence large variance) in human-annotated labels. In this paper, we present a framework that can learn high-level feature representations with a limited amount of data, by first modelling their corresponding quantifiable low-level attributes. We refer to our proposed framework as Music FaderNets, which is inspired by the fact that low-level attributes can be continuously manipulated by separate "sliding faders" through feature disentanglement and latent regularization techniques. High-level features are then inferred from the low-level representations through semi-supervised clustering using Gaussian Mixture Variational Autoencoders (GM-VAEs). Using arousal as an example of a high-level feature, we show that the "faders" of our model are disentangled and change linearly w.r.t. the modelled low-level attributes of the generated output music. Furthermore, we demonstrate that the model successfully learns the intrinsic relationship between arousal and its corresponding low-level attributes (rhythm and note density), with only 1% of the training set being labelled. Finally, using the learnt high-level feature representations, we explore the application of our framework in style transfer tasks across different arousal states. The effectiveness of this approach is verified through a subjective listening test.
References in corpus (18)
- Auto-Encoding Variational Bayes
- Conditional Generative Adversarial Nets
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
- Disentangled Sequential Autoencoder
- Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
- MorpheuS: generating structured music with constrained patterns and tension
- MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer
- On Variational Learning of Controllable Representations for Text without Supervision
- A Tutorial on Deep Latent Variable Models of Natural Language
- Deep Music Analogy Via Latent Representation Disentanglement
- Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models
- Music Style Transfer: A Position Paper
- Semi-Supervised Generative Modeling for Controllable Speech Synthesis
- Learning Disentangled Representations for Timber and Pitch in Music Audio
- Musical Composition Style Transfer via Disentangled Timbre Representations
- Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders