WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN

Abstract

We present a deep neural network based singing voice synthesizer, inspired by the Deep Convolutions Generative Adversarial Networks (DCGAN) architecture and optimized using the Wasserstein-GAN algorithm. We use vocoder parameters for acoustic modelling, to separate the influence of pitch and timbre. This facilitates the modelling of the large variability of pitch in the singing voice. Our network takes a block of consecutive frame-wise linguistic and fundamental frequency features, along with global singer identity as input and outputs vocoder features, corresponding to the block of features. This block-wise approach, along with the training methodology allows us to model temporal dependencies within the features of the input block. For inference, sequential blocks are concatenated using an overlap-add procedure. We show that the performance of our model is competitive with regards to the state-of-the-art and the original sample using objective metrics and a subjective listening test. We also present examples of the synthesis on a supplementary website and the source code via GitHub.

References in corpus (5)

• Conditional Generative Adversarial Nets
• WaveNet: A Generative Model for Raw Audio
• GANSynth: Adversarial Neural Audio Synthesis
• Adversarial Audio Synthesis
• Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation

Cited by in corpus (14)

• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• DDSP: Differentiable Digital Signal Processing
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis
• MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
• Latent Space Explorations of Singing Voice Synthesis using DDSP
• Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web
• Peking Opera Synthesis via Duration Informed Attention Network
• Synthesising Expressiveness in Peking Opera via Duration Informed Attention Network
• Speech Synthesis as Augmentation for Low-Resource ASR
• A Deep Learning Based Analysis-Synthesis Framework For Unison Singing
• Using a Bi-directional LSTM Model with Attention Mechanism trained on MIDI Data for Generating Unique Music
• Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss