Jukebox: A Generative Model for Music

Abstract

We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples at https://jukebox.openai.com, along with model weights and code at https://github.com/openai/jukebox

References in corpus (7)

• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• MelNet: A Generative Model for Audio in the Frequency Domain
• Fixup Initialization: Residual Learning Without Normalization
• Counterpoint by Convolution

Cited by in corpus (42)

• Learning Transferable Visual Models From Natural Language Supervision
• Diffusion Models Beat GANs on Image Synthesis
• Evaluating Large Language Models Trained on Code
• Zero-Shot Text-to-Image Generation
• Scaling Laws for Autoregressive Generative Modeling
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• POP909: A Pop-song Dataset for Music Arrangement Generation
• ATISS: Autoregressive Transformers for Indoor Scene Synthesis
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Stochastic Restoration of Heavily Compressed Musical Audio using Generative Adversarial Networks
• Predicting Video with VQVAE
• Codified audio language modeling learns useful representations for music information retrieval
• Taming Visually Guided Sound Generation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Controllable deep melody generation via hierarchical music structure representation
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Tiny Transformers for Environmental Sound Classification at the Edge
• Variable-rate discrete representation learning
• Generating Images with Sparse Representations
• Dance2Music: Automatic Dance-driven Music Generation
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• MTCRNN: A multi-scale RNN for directed audio texture synthesis
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web
• AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
• Melody Classifier with Stacked-LSTM
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Exploring Disentanglement with Multilingual and Monolingual VQ-VAE
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Hierarchical Timbre-Painting and Articulation Generation
• Actions Speak Louder than Listening: Evaluating Music Style Transfer based on Editing Experience
• Unsupervised Source Separation By Steering Pretrained Music Models
• Ownership and Creativity in Generative Models
• Musical Speech: A Transformer-based Composition Tool
• BumbleBee: A Transformer for Music
• PocketVAE: A Two-step Model for Groove Generation and Control