Musical Speech: A Transformer-based Composition Tool
arXiv:2108.01043
Abstract
In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline -- which does not require a paired dataset for training -- through examples of music created by musicians making use of our tool.
arXiv commentNeurIPS 2020 Demonstration Track; extended for PMLR
References in corpus (9)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
- The challenge of realistic music generation: modelling raw audio at scale
- MelNet: A Generative Model for Audio in the Frequency Domain
- Jukebox: A Generative Model for Music
- MorpheuS: generating structured music with constrained patterns and tension
- Counterpoint by Convolution
- A Universal Music Translation Network
- Performing Structured Improvisations with pre-trained Deep Learning Models