A Spectral Energy Distance for Parallel Speech Synthesis

Abstract

Speech synthesis is an important practical generative modeling problem that has seen great progress over the last few years, with likelihood-based autoregressive neural models now outperforming traditional concatenative systems. A downside of such autoregressive models is that they require executing tens of thousands of sequential operations per second of generated audio, making them ill-suited for deployment on specialized deep learning hardware. Here, we propose a new learning method that allows us to train highly parallel models of speech, without requiring access to an analytical likelihood function. Our approach is based on a generalized energy distance between the distributions of the generated and real audio. This spectral energy distance is a proper scoring rule with respect to the distribution over magnitude-spectrograms of the generated waveform audio and offers statistical consistency guarantees. The distance can be calculated from minibatches without bias, and does not involve adversarial learning, yielding a stable and consistent method for training implicit generative models. Empirically, we achieve state-of-the-art generation quality among implicit generative models, as judged by the recently-proposed cFDSD metric. When combining our method with adversarial techniques, we also improve upon the recently-proposed GAN-TTS model in terms of Mean Opinion Score as judged by trained human evaluators.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Density estimation using Real NVP
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Generative Moment Matching Networks
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• Efficient Neural Audio Synthesis
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• WaveFlow: A Compact Flow-based Model for Raw Audio
• FloWaveNet : A Generative Flow for Raw Audio

Cited by in corpus (4)

• WaveGrad: Estimating Gradients for Waveform Generation
• Review of end-to-end speech synthesis technology based on deep learning
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus