MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Abstract

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

Cited by in corpus (12)

• Jukebox: A Generative Model for Music
• DiscreTalk: Text-to-Speech as a Machine Translation Problem
• HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
• Adversarial representation learning for private speech generation
• HooliGAN: Robust, High Quality Neural Vocoding
• Controllable neural text-to-speech synthesis using intuitive prosodic features
• VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• Speech-to-Singing Conversion based on Boundary Equilibrium GAN
• Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent Dilated Convolution Model for Parametric Speech Generation
• Gamma Boltzmann Machine for Simultaneously Modeling Linear- and Log-amplitude Spectra