Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization

Abstract

In a recent paper, we have presented a generative adversarial network (GAN)-based model for unconditional generation of the mel-spectrograms of singing voices. As the generator of the model is designed to take a variable-length sequence of noise vectors as input, it can generate mel-spectrograms of variable length. However, our previous listening test shows that the quality of the generated audio leaves room for improvement. The present paper extends and expands that previous work in the following aspects. First, we employ a hierarchical architecture in the generator to induce some structure in the temporal dimension. Second, we introduce a cycle regularization mechanism to the generator to avoid mode collapse. Third, we evaluate the performance of the new model not only for generating singing voices, but also for generating speech voices. Evaluation result shows that new model outperforms the prior one both objectively and subjectively. We also employ the model to unconditionally generate sequences of piano and violin music and find the result promising. Audio examples, as well as the code for implementing our model, will be publicly available online upon paper publication.

References in corpus (10)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Conditional Generative Adversarial Nets
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• GANSynth: Adversarial Neural Audio Synthesis
• MelNet: A Generative Model for Audio in the Frequency Domain
• DDSP: Differentiable Digital Signal Processing
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
• Unsupervised Speech Decomposition via Triple Information Bottleneck
• Score and Lyrics-Free Singing Voice Generation
• Speech-to-Singing Conversion in an Encoder-Decoder Framework

Cited by in corpus (7)

• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• DPN-GAN: Inducing Periodic Activations in Generative Adversarial Networks for High-Fidelity Audio Synthesis
• RWCP-SSD-Onomatopoeia: Onomatopoeic Word Dataset for Environmental Sound Synthesis
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN