A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
arXiv:2011.06801
Abstract
The utilization of deep learning techniques in generating various contents (such as image, text, etc.) has become a trend. Especially music, the topic of this paper, has attracted widespread attention of countless researchers.The whole process of producing music can be divided into three stages, corresponding to the three levels of music generation: score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directly. Previous surveys have explored the network models employed in the field of automatic music generation. However, the development history, the model evolution, as well as the pros and cons of same music generation task have not been clearly illustrated. This paper attempts to provide an overview of various composition tasks under different music generation levels, covering most of the currently popular music generation tasks using deep learning. In addition, we summarize the datasets suitable for diverse tasks, discuss the music representations, the evaluation methods as well as the challenges under different levels, and finally point out several future directions.
96 pages,this is a draft
References in corpus (70)
- Distributed Representations of Words and Phrases and their Compositionality
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Conditional Generative Adversarial Nets
- From Frequency to Meaning: Vector Space Models of Semantics
- Neural Discrete Representation Learning
- Conditional Image Generation with PixelCNN Decoders
- Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
- Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
- C-RNN-GAN: Continuous recurrent neural networks with adversarial training
- SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
- Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
- MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
- Audio Spectrogram Representations for Processing with Convolutional Neural Networks
- The challenge of realistic music generation: modelling raw audio at scale
- MelNet: A Generative Model for Audio in the Frequency Domain
- Jukebox: A Generative Model for Music
- High Fidelity Speech Synthesis with Adversarial Networks
- Adversarial Audio Synthesis
- Counterpoint by Convolution
- DDSP: Differentiable Digital Signal Processing
- Music transcription modelling and composition using deep learning
- Text-based LSTM networks for Automatic Music Composition
- GACELA -- A generative adversarial context encoder for long audio inpainting
- WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
- POP909: A Pop-song Dataset for Music Arrangement Generation
- Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints
- Latent Normalizing Flows for Discrete Sequences
- Learning to Groove with Inverse Sequence Transformations
- MMM : Exploring Conditional Multi-Track Music Generation with the Transformer
- Learning a Latent Space of Multitrack Measures
- Song From PI: A Musically Plausible Network for Pop Music Generation
- Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
- MGGAN: Solving Mode Collapse using Manifold Guided Training
- Introducing SPAIN (SParse Audio INpainter)
- Singing voice synthesis based on convolutional neural networks
- Deep Long Audio Inpainting
- Interactive Music Generation with Positional Constraints using Anticipation-RNNs
- PIANOTREE VAE: Structured Representation Learning for Polyphonic Music
- XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System
- Neural Translation of Musical Style
- ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
- Score and Lyrics-Free Singing Voice Generation
- Modulated Variational auto-Encoders for many-to-many musical timbre transfer
- High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
- DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion System
- Learning Disentangled Representations for Timber and Pitch in Music Audio
- Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
- VaPar Synth -- A Variational Parametric Model for Audio Synthesis
- From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends
- Time Domain Neural Audio Style Transfer
- Generating Albums with SampleRNN to Imitate Metal, Rock, and Punk Bands
- Explicitly Conditioned Melody Generation: A Case Study with Interdependent RNNs
- Learning Singing From Speech
- NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting
- Encoding Musical Style with Transformer Autoencoders
- Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer
- Learning Style-Aware Symbolic Music Representations by Adversarial Autoencoders
- A database linking piano and orchestral MIDI scores with application to automatic projective orchestration
- Conditioning Deep Generative Raw Audio Models for Structured Automatic Music
- Automatic Melody Harmonization with Triad Chords: A Comparative Study
- BachProp: Learning to Compose Music in Multiple Styles
- Inspecting and Interacting with Meaningful Music Representations using VAE
- Music Generation with Deep Learning
- Live Orchestral Piano, a system for real-time orchestral music generation
- Peking Opera Synthesis via Duration Informed Attention Network
- Generative Modelling for Controllable Audio Synthesis of Expressive Piano Performance
- DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
- DeepDrum: An Adaptive Conditional Neural Network
- MusicTM-Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio
Cited by in corpus (9)
- GWA: A Large High-Quality Acoustic Dataset for Audio Processing
- BERT-like Pre-training for Symbolic Piano Music Classification Tasks
- Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography
- MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
- DVGAN: Stabilize Wasserstein GAN training for time-domain Gravitational Wave physics
- Score Transformer: Generating Musical Score from Note-level Representation
- Enhanced Memory Network: The novel network structure for Symbolic Music Generation
- Symbolic Music Loop Generation with VQ-VAE
- WakaVT: A Sequential Variational Transformer for Waka Generation