MusicLM: Generating Music From Text

Abstract

We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.

arXiv commentSupplementary material at https://google-research.github.io/seanet/musiclm/examples and https://kaggle.com/datasets/googleai/musiccaps

Cited by in corpus (14)

• Generative AI
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data Producers
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• SingSong: Generating musical accompaniments from singing
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
• Learning to Taste: A Multimodal Wine Dataset
• Example-Based Framework for Perceptually Guided Audio Texture Generation
• Benchmarking Music Generation Models and Metrics via Human Preference Studies
• A Multimodal Symphony: Integrating Taste and Sound through Generative AI
• M6(GPT)3: Generating Multitrack Modifiable Multi-Minute MIDI Music from Text using Genetic algorithms, Probabilistic methods and GPT Models in any Progression and Time Signature
• Six misconceptions about large language models: A minimal model and diagnostic taxonomy