Jukebox: A Generative Model for Music

Abstract

We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples at https://jukebox.openai.com, along with model weights and code at https://github.com/openai/jukebox

References in corpus (21)

• Auto-Encoding Variational Bayes
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Self-Attention Generative Adversarial Networks
• Conditional Image Generation with PixelCNN Decoders
• Variational Inference with Normalizing Flows
• Dota 2 with Large Scale Deep Reinforcement Learning
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Generating Long Sequences with Sparse Transformers
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Axial Attention in Multidimensional Transformers
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
• Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• The challenge of realistic music generation: modelling raw audio at scale
• MelNet: A Generative Model for Audio in the Frequency Domain
• Fixup Initialization: Residual Learning Without Normalization
• Counterpoint by Convolution
• Hierarchical Autoregressive Image Models with Auxiliary Decoders

Cited by in corpus (71)

• Learning Transferable Visual Models From Natural Language Supervision
• Diffusion Models Beat GANs on Image Synthesis
• Evaluating Large Language Models Trained on Code
• Zero-Shot Text-to-Image Generation
• Scaling Laws for Autoregressive Generative Modeling
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Deep Learning for Procedural Content Generation
• Review: Deep Learning in Electron Microscopy
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• POP909: A Pop-song Dataset for Music Arrangement Generation
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• ATISS: Autoregressive Transformers for Indoor Scene Synthesis
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Creativity and Machine Learning: A Survey
• Computer-Aided Design as Language
• Simple and Effective VAE Training with Calibrated Decoders
• Audio Transformers
• A Simple Convergence Proof of Adam and Adagrad
• Stochastic Restoration of Heavily Compressed Musical Audio using Generative Adversarial Networks
• Predictive Coding: a Theoretical and Experimental Review
• Predicting Video with VQVAE
• Codified audio language modeling learns useful representations for music information retrieval
• A Spectral Energy Distance for Parallel Speech Synthesis
• Taming Visually Guided Sound Generation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Controllable deep melody generation via hierarchical music structure representation
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Score-based Generative Modeling in Latent Space
• Tiny Transformers for Environmental Sound Classification at the Edge
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Variable-rate discrete representation learning
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• Symbolic Music Generation with Diffusion Models
• Generating Images with Sparse Representations
• Dance2Music: Automatic Dance-driven Music Generation
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Generative Models as a Data Source for Multiview Representation Learning
• Transfer Learning with Jukebox for Music Source Separation
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• MTCRNN: A multi-scale RNN for directed audio texture synthesis
• Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web
• MT3: Multi-Task Multitrack Music Transcription
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• Melody Classifier with Stacked-LSTM
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Exploring Disentanglement with Multilingual and Monolingual VQ-VAE
• Unsupervised Source Separation via Bayesian Inference in the Latent Domain
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Hierarchical Timbre-Painting and Articulation Generation
• PocketVAE: A Two-step Model for Groove Generation and Control
• Musical Speech: A Transformer-based Composition Tool
• Actions Speak Louder than Listening: Evaluating Music Style Transfer based on Editing Experience
• Unsupervised Source Separation By Steering Pretrained Music Models
• BumbleBee: A Transformer for Music
• Symbolic Music Loop Generation with VQ-VAE
• Ownership and Creativity in Generative Models
• Generative Deep Learning for Virtuosic Classical Music: Generative Adversarial Networks as Renowned Composers