On granularity of prosodic representations in expressive text-to-speech

Abstract

In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Word-level representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.

arXiv commentAccepted to IEEE SLT 2022

References in corpus (8)

• WaveNet: A Generative Model for Raw Audio
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
• Creating New Voices using Normalizing Flows
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis