Learning to Groove with Inverse Sequence Transformations

Abstract

We explore models for translating abstract musical ideas (scores, rhythms) into expressive performances using Seq2Seq and recurrent Variational Information Bottleneck (VIB) models. Though Seq2Seq models usually require painstakingly aligned corpora, we show that it is possible to adapt an approach from the Generative Adversarial Network (GAN) literature (e.g. Pix2Pix (Isola et al., 2017) and Vid2Vid (Wang et al. 2018a)) to sequences, creating large volumes of paired data by performing simple transformations and training generative models to plausibly invert these transformations. Music, and drumming in particular, provides a strong test case for this approach because many common transformations (quantization, removing voices) have clear semantics, and models for learning to invert them have real-world applications. Focusing on the case of drum set players, we create and release a new dataset for this purpose, containing over 13 hours of recordings by professional drummers aligned with fine-grained timing and dynamics information. We also explore some of the creative potential of these models, including demonstrating improvements on state-of-the-art methods for Humanization (instantiating a performance from a musical score).

arXiv commentBlog post and links: https://g.co/magenta/groovae

Cited by in corpus (14)

• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• An Interdisciplinary Review of Music Performance Analysis
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset
• Encoding Musical Style with Transformer Autoencoders
• Can GAN originate new electronic dance music genres? -- Generating novel rhythm patterns using GAN with Genre Ambiguity Loss
• Towards democratizing music production with AI-Design of Variational Autoencoder-based Rhythm Generator as a DAW plugin
• Downbeat Tracking with Tempo-Invariant Convolutional Neural Networks
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• The GigaMIDI Dataset with Features for Expressive Music Performance Detection
• Artificial Neural Networks Jamming on the Beat
• PocketVAE: A Two-step Model for Groove Generation and Control
• Symbolic Music Loop Generation with VQ-VAE