Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE

Abstract

In this paper, we propose a novel unsupervised text-to-speech acoustic model training scheme, named UTTS, which does not require text-audio pairs. UTTS is a multi-speaker speech synthesizer that supports zero-shot voice cloning, it is developed from a perspective of disentangled speech representation learning. The framework offers a flexible choice of a speaker's duration model, timbre feature (identity) and content for TTS inference. We leverage recent advancements in self-supervised speech representation learning as well as speech synthesis front-end techniques for system development. Specifically, we employ our recently formulated Conditional Disentangled Sequential Variational Auto-encoder (C-DSVAE) as the backbone UTTS AM, which offers well-structured content representations given unsupervised alignment (UA) as condition during training. For UTTS inference, we utilize a lexicon to map input text to the phoneme sequence, which is expanded to the frame-level forced alignment (FA) with a speaker-dependent duration model. Then, we develop an alignment mapping module that converts FA to UA. Finally, the C-DSVAE, serving as the self-supervised TTS AM, takes the predicted UA and a target speaker embedding to generate the mel spectrogram, which is ultimately converted to waveform with a neural vocoder. We show how our method enables speech synthesis without using a paired TTS corpus in AM development stage. Experiments demonstrate that UTTS can synthesize speech of high naturalness and intelligibility measured by human and objective evaluations. Audio samples are available at our demo page https://neurtts.github.io/utts\_demo/.

IEEE/ACM Transactions on Audio, Speech, and Language Processing ( Volume: 31)

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Denoising Diffusion Probabilistic Models
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• Disentangled Sequential Autoencoder
• Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning
• C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verification
• Disentangled Recurrent Wasserstein Autoencoder
• Contrastively Disentangled Sequential Variational Autoencoder