MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Abstract

Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative adversarial networks (GANs). The three models, which differ in the underlying assumptions and accordingly the network architectures, are referred to as the jamming model, the composer model and the hybrid model. We trained the proposed models on a dataset of over one hundred thousand bars of rock music and applied them to generate piano-rolls of five tracks: bass, drums, guitar, piano and strings. A few intra-track and inter-track objective metrics are also proposed to evaluate the generative results, in addition to a subjective user study. We show that our models can generate coherent music of four bars right from scratch (i.e. without human inputs). We also extend our models to human-AI cooperative music generation: given a specific track composed by human, we can generate four additional tracks to accompany it. All code, the dataset and the rendered audio samples are available at https://salu133445.github.io/musegan/ .

to appear at AAAI 2018

Cited by in corpus (60)

• IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detection
• Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions
• Jukebox: A Generative Model for Music
• Video Background Music Generation with Controllable Music Transformer
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Copyright in Generative Deep Learning
• QuGAN: A Quantum State Fidelity based Generative Adversarial Network
• POP909: A Pop-song Dataset for Music Arrangement Generation
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
• Creativity and Machine Learning: A Survey
• The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures
• Learning a Latent Space of Multitrack Measures
• LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
• MusPy: A Toolkit for Symbolic Music Generation
• Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm
• DAWSON: A Domain Adaptive Few Shot Generation Framework
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
• Adversarial Learning for Improved Onsets and Frames Music Transcription
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• Controllable deep melody generation via hierarchical music structure representation
• A Hierarchical Recurrent Neural Network for Symbolic Melody Generation
• Explicitly Conditioned Melody Generation: A Case Study with Interdependent RNNs
• MIDI-Sandwich2: RNN-based Hierarchical Multi-modal Fusion Generation VAE networks for multi-track symbolic music generation
• Training Generative Adversarial Networks with Binary Neurons by End-to-end Backpropagation
• GANwriting: Content-Conditioned Generation of Styled Handwritten Word Images
• Artificial Musical Intelligence: A Survey
• PopMAG: Pop Music Accompaniment Generation
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Musical Composition Style Transfer via Disentangled Timbre Representations
• Generating Music with a Self-Correcting Non-Chronological Autoregressive Model
• SurpriseNet: Melody Harmonization Conditioning on User-controlled Surprise Contours
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Towards Better Long-range Time Series Forecasting using Generative Adversarial Networks
• Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network
• AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• Automatic Neural Lyrics and Melody Composition
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• Manifold Interpolation for Large-Scale Multi-Objective Optimization via Generative Adversarial Networks
• Anticipation in collaborative music performance using fuzzy systems: a case study
• Melody Classifier with Stacked-LSTM
• Enhanced Memory Network: The novel network structure for Symbolic Music Generation
• Melody Structure Transfer Network: Generating Music with Separable Self-Attention
• Symbolic Music Playing Techniques Generation as a Tagging Problem
• Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music
• MIDI-Sandwich: Multi-model Multi-task Hierarchical Conditional VAE-GAN networks for Symbolic Single-track Music Generation
• A novel dataset for the identification of computer generated melodies in the CSMT challenge
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Exploring Inherent Properties of the Monophonic Melody of Songs
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Dual-track Music Generation using Deep Learning
• On the Veracity of Cyber Intrusion Alerts Synthesized by Generative Adversarial Networks
• Temporal Pattern Attention for Multivariate Time Series Forecasting
• Harmonic Mean Point Processes: Proportional Rate Error Minimization for Obtundation Prediction
• Subspace Capsule Network
• BumbleBee: A Transformer for Music
• Symbolic Music Loop Generation with VQ-VAE
• Music Embedding: A Tool for Incorporating Music Theory into Computational Music Applications
• RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning