Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Abstract

Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

arXiv commentExamples available at https://goo.gl/magenta/maestro-examples

Cited by in corpus (40)

• Jukebox: A Generative Model for Music
• Video Background Music Generation with Controllable Music Transformer
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• GACELA -- A generative adversarial context encoder for long audio inpainting
• POP909: A Pop-song Dataset for Music Arrangement Generation
• An Interdisciplinary Review of Music Performance Analysis
• Toward Interpretable Music Tagging with Self-Attention
• Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
• LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
• MusPy: A Toolkit for Symbolic Music Generation
• Non-Local Musical Statistics as Guides for Audio-to-Score Piano Transcription
• Explaining Deep Neural Networks
• Adversarial Learning for Improved Onsets and Frames Music Transcription
• BERT-like Pre-training for Symbolic Piano Music Classification Tasks
• High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
• MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
• Foley Music: Learning to Generate Music from Videos
• Symbolic Music Generation with Diffusion Models
• Encoding Musical Style with Transformer Autoencoders
• Polyphonic Piano Transcription Using Autoregressive Multi-State Note Model
• Towards Interpretable Polyphonic Transcription with Invertible Neural Networks
• Deep Composer Classification Using Symbolic Representation
• Large-Scale MIDI-based Composer Classification
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Generative Modelling for Controllable Audio Synthesis of Expressive Piano Performance
• Deep Layered Learning in MIR
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data
• The GigaMIDI Dataset with Features for Expressive Music Performance Detection
• Data Cleansing with Contrastive Learning for Vocal Note Event Annotations
• Text-to-Speech Synthesis Techniques for MIDI-to-Audio Synthesis
• Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability
• Optimal Embedding Calibration for Symbolic Music Similarity
• Computer Assisted Composition in Continuous Time
• Rethinking Evaluation Methodology for Audio-to-Score Alignment
• The MIDI Degradation Toolkit: Symbolic Music Augmentation and Correction
• Using a Bi-directional LSTM Model with Attention Mechanism trained on MIDI Data for Generating Unique Music
• Latent Time-Adaptive Drift-Diffusion Model
• Musical Speech: A Transformer-based Composition Tool
• Learning To Generate Piano Music With Sustain Pedals