A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

Abstract

The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at http://g.co/magenta/musicvae-code.

arXiv commentICML Camera Ready Version (w/ fixed typos)

References in corpus (18)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Conditional Image Generation with PixelCNN Decoders
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• A Recurrent Latent Variable Model for Sequential Data
• A Neural Representation of Sketch Drawings
• Semi-supervised Sequence Learning
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Variational Lossy Autoencoder
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Variational Recurrent Auto-Encoders
• PixelVAE: A Latent Variable Model for Natural Images
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Fixing a Broken ELBO
• Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

Cited by in corpus (67)

• Dynamical Variational Autoencoders: A Comprehensive Review
• Rethinking CNN Models for Audio Classification
• Jukebox: A Generative Model for Music
• Video Background Music Generation with Controllable Music Transformer
• A Deep Generative Model for Graph Layout
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
• Learning a Latent Space of Multitrack Measures
• Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
• Deep Music Analogy Via Latent Representation Disentanglement
• Structure-Enhanced Pop Music Generation via Harmony-Aware Learning
• Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI
• Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
• Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions
• Autoencoding Low-Resolution MRI for Semantically Smooth Interpolation of Anisotropic MRI
• A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling
• Learning Disentangled Representations for Timber and Pitch in Music Audio
• Latent Translation: Crossing Modalities by Bridging Generative Models
• Controllable deep melody generation via hierarchical music structure representation
• Variational Laplace Autoencoders
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends
• Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
• Explicitly Conditioned Melody Generation: A Case Study with Interdependent RNNs
• A Hierarchical Recurrent Neural Network for Symbolic Melody Generation
• Foley Music: Learning to Generate Music from Videos
• Symbolic Music Generation with Diffusion Models
• Improving Botnet Detection with Recurrent Neural Network and Transfer Learning
• Encoding Musical Style with Transformer Autoencoders
• Concept-Oriented Deep Learning: Generative Concept Representations
• Learning Style-Aware Symbolic Music Representations by Adversarial Autoencoders
• Generative Melody Composition with Human-in-the-Loop Bayesian Optimization
• Botnet Detection Using Recurrent Variational Autoencoder
• PopMAG: Pop Music Accompaniment Generation
• Coupled Recurrent Models for Polyphonic Music Composition
• Mechanisms of Artistic Creativity in Deep Learning Neural Networks
• Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning
• Log Message Anomaly Detection and Classification Using Auto-B/LSTM and Auto-GRU
• Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network
• Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents
• Feasibility of Modeling Orthogonal Frequency-Division Multiplexing Communication Signals with Unsupervised Generative Adversarial Networks
• Coevo: a collaborative design platform with artificial agents
• Neural Melody Composition from Lyrics
• Towards democratizing music production with AI-Design of Variational Autoencoder-based Rhythm Generator as a DAW plugin
• Personalized Popular Music Generation Using Imitation and Structure
• Exploring Conditioning for Generative Music Systems with Human-Interpretable Controls
• Latent Variable Modeling for Generative Concept Representations and Deep Generative Models
• Deep Predictive Models in Interactive Music
• Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
• Artificial Neural Networks Jamming on the Beat
• Performing Structured Improvisations with pre-trained Deep Learning Models
• A novel dataset for the identification of computer generated melodies in the CSMT challenge
• Signal-domain representation of symbolic music for learning embedding spaces
• PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network
• Modeling Coherency in Generated Emails by Leveraging Deep Neural Learners
• BumbleBee: A Transformer for Music
• Symbolic Music Loop Generation with VQ-VAE
• Using Shapley Values and Variational Autoencoders to Explain Predictive Models with Dependent Mixed Features
• Markov-switching State Space Models for Uncovering Musical Interpretation
• Modeling Melodic Feature Dependency with Modularized Variational Auto-Encoder
• Harmonic Recomposition using Conditional Autoregressive Modeling
• Formal Verification of Long Short-Term Memory based Audio Classifiers: A Star based Approach