WavThruVec: Latent speech representation as intermediate features for neural speech synthesis

Abstract

Recent advances in neural text-to-speech research have been dominated by two-stage pipelines utilizing low-level intermediate speech representation such as mel-spectrograms. However, such predetermined features are fundamentally limited, because they do not allow to exploit the full potential of a data-driven approach through learning hidden representations. For this reason, several end-to-end methods have been proposed. However, such models are harder to train and require a large number of high-quality recordings with transcriptions. Here, we propose WavThruVec - a two-stage architecture that resolves the bottleneck by using high-dimensional Wav2Vec 2.0 embeddings as intermediate speech representation. Since these hidden activations provide high-level linguistic features, they are more robust to noise. That allows us to utilize annotated speech datasets of a lower quality to train the first-stage module. At the same time, the second-stage component can be trained on large-scale untranscribed audio corpora, as Wav2Vec 2.0 embeddings are already time-aligned. This results in an increased generalization capability to out-of-vocabulary words, as well as to a better generalization to unseen speakers. We show that the proposed model not only matches the quality of state-of-the-art neural models, but also presents useful properties enabling tasks like voice conversion or zero-shot synthesis.

Accepted to INTERSPEECH 2022. Audio samples are available at: https://charactr-platform.github.io/WavThruVec/

References in corpus (8)

• WaveNet: A Generative Model for Raw Audio
• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• A Survey on Neural Speech Synthesis

Cited by in corpus (3)

• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing
• Emotion-Coherent Speech Data Augmentation and Self-Supervised Contrastive Style Training for Enhancing Kids's Story Speech Synthesis