High Fidelity Speech Synthesis with Adversarial Networks

Abstract

Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at https://storage.googleapis.com/deepmind-media/research/abstract.wav.

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A Learned Representation For Artistic Style
• Adversarially Learned Inference
• Deep Voice: Real-time Neural Text-to-Speech
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Adversarial Video Generation on Complex Datasets
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• MelNet: A Generative Model for Audio in the Frequency Domain
• Adversarial Audio Synthesis
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms

Cited by in corpus (22)

• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• WaveFlow: A Compact Flow-based Model for Raw Audio
• GACELA -- A generative adversarial context encoder for long audio inpainting
• CorrGAN: Sampling Realistic Financial Correlation Matrices Using Generative Adversarial Networks
• ESPnet2-TTS: Extending the Edge of TTS Research
• Quasi-Periodic Parallel WaveGAN: A Non-autoregressive Raw Waveform Generative Model with Pitch-dependent Dilated Convolution Neural Network
• A Spectral Energy Distance for Parallel Speech Synthesis
• HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks
• Non Gaussian Denoising Diffusion Models
• Score and Lyrics-Free Singing Voice Generation
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizer
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Denoising Diffusion Gamma Models
• DPN-GAN: Inducing Periodic Activations in Generative Adversarial Networks for High-Fidelity Audio Synthesis
• Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis
• Mathematical Vocoder Algorithm : Modified Spectral Inversion for Efficient Neural Speech Synthesis
• AI based Presentation Creator With Customized Audio Content Delivery
• Generating Diverse Realistic Laughter for Interactive Art