Parallel Synthesis for Autoregressive Speech Generation
arXiv:2204.11806 · doi:10.1109/TASLP.2023.3301212
Abstract
Autoregressive neural vocoders have achieved outstanding performance in speech synthesis tasks such as text-to-speech and voice conversion. An autoregressive vocoder predicts a sample at some time step conditioned on those at previous time steps. Though it synthesizes natural human speech, the iterative generation inevitably makes the synthesis time proportional to the utterance length, leading to low efficiency. Many works were dedicated to generating the whole speech sequence in parallel and proposed GAN-based, flow-based, and score-based vocoders. This paper proposed a new thought for the autoregressive generation. Instead of iteratively predicting samples in a time sequence, the proposed model performs frequency-wise autoregressive generation (FAR) and bit-wise autoregressive generation (BAR) to synthesize speech. In FAR, a speech utterance is split into frequency subbands, and a subband is generated conditioned on the previously generated one. Similarly, in BAR, an 8-bit quantized signal is generated iteratively from the first bit. By redesigning the autoregressive method to compute in domains other than the time domain, the number of iterations in the proposed model is no longer proportional to the utterance length but to the number of subbands/bits, significantly increasing inference efficiency. Besides, a post-filter is employed to sample signals from output posteriors; its training objective is designed based on the characteristics of the proposed methods. Experimental results show that the proposed model can synthesize speech faster than real-time without GPU acceleration. Compared with baseline vocoders, the proposed model achieves better MUSHRA results and shows good generalization ability for unseen speakers and 44 kHz speech.
IEEE/ACM Transactions on Audio, Speech, and Language Processing
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- WaveNet: A Generative Model for Raw Audio
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- Non-Autoregressive Neural Machine Translation
- Parallel WaveNet: Fast High-Fidelity Speech Synthesis
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
- Unsupervised speech representation learning using WaveNet autoencoders
- AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
- Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
- Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension
- PolyGen: An Autoregressive Generative Model of 3D Meshes
- WaveFlow: A Compact Flow-based Model for Raw Audio
- Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
- Voice Conversion for Whispered Speech Synthesis
- Towards Robust Neural Vocoding for Speech Generation: A Survey
- TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
- SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis