The challenge of realistic music generation: modelling raw audio at scale

Abstract

Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.

arXiv comment13 pages, 2 figures, submitted to NIPS 2018

Cited by in corpus (36)

• Generating Long Sequences with Sparse Transformers
• NVAE: A Deep Hierarchical Variational Autoencoder
• MelNet: A Generative Model for Audio in the Frequency Domain
• Jukebox: A Generative Model for Music
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• WaveFlow: A Compact Flow-based Model for Raw Audio
• GACELA -- A generative adversarial context encoder for long audio inpainting
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• Self-Supervised VQ-VAE for One-Shot Music Style Transfer
• LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
• Quasi-Periodic WaveNet: An Autoregressive Raw Waveform Generative Model with Pitch-dependent Dilated Convolution Neural Network
• Predicting Video with VQVAE
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Codified audio language modeling learns useful representations for music information retrieval
• Unsupervised Speech Recognition
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Self-training and Pre-training are Complementary for Speech Recognition
• Variable-rate discrete representation learning
• MTCRNN: A multi-scale RNN for directed audio texture synthesis
• Artificial Musical Intelligence: A Survey
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Learning Neurosymbolic Generative Models via Program Synthesis
• Extractive Summary as Discrete Latent Variables
• Exploring Conditioning for Generative Music Systems with Human-Interpretable Controls
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• Normalizing Flows with Multi-Scale Autoregressive Priors
• Using a Bi-directional LSTM Model with Attention Mechanism trained on MIDI Data for Generating Unique Music
• Transferring neural speech waveform synthesizers to musical instrument sounds generation
• PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network
• Musical Speech: A Transformer-based Composition Tool
• Depthwise Discrete Representation Learning
• A Comparison of Discrete Latent Variable Models for Speech Representation Learning
• SchrödingeRNN: Generative Modeling of Raw Audio as a Continuously Observed Quantum State
• DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
• PixelPyramids: Exact Inference Models from Lossless Image Pyramids