CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network

Abstract

The prosodic aspects of speech signals produced by current text-to-speech systems are typically averaged over training material, and as such lack the variety and liveliness found in natural speech. To avoid monotony and averaged prosody contours, it is desirable to have a way of modeling the variation in the prosodic aspects of speech, so audio signals can be synthesized in multiple ways for a given text. We present a new, hierarchically structured conditional variational autoencoder to generate prosodic features (fundamental frequency, energy and duration) suitable for use with a vocoder or a generative model like WaveNet. At inference time, an embedding representing the prosody of a sentence may be sampled from the variational layer to allow for prosodic variation. To efficiently capture the hierarchical nature of the linguistic input (words, syllables and phones), both the encoder and decoder parts of the auto-encoder are hierarchical, in line with the linguistic structure, with layers being clocked dynamically at the respective rates. We show in our experiments that our dynamic hierarchical network outperforms a non-hierarchical state-of-the-art baseline, and, additionally, that prosody transfer across sentences is possible by employing the prosody embedding of one sentence to generate the speech signal of another.

References in corpus (5)

• Auto-Encoding Variational Bayes
• Exploring the Limits of Language Modeling
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Hierarchical Generative Modeling for Controllable Speech Synthesis

Cited by in corpus (21)

• Emotion Intensity and its Control for Emotional Voice Conversion
• Improving Universal Sound Separation Using Sound Classification
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• Unsupervised Speech Decomposition via Triple Information Bottleneck
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• Review of end-to-end speech synthesis technology based on deep learning
• Expressive TTS Training with Frame and Style Reconstruction Loss
• Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
• Global Rhythm Style Transfer Without Text Transcriptions
• Controllable neural text-to-speech synthesis using intuitive prosodic features
• MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
• Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• Applying the Information Bottleneck Principle to Prosodic Representation Learning
• Hierarchical Context-Aware Transformers for Non-Autoregressive Text to Speech