Controllable speech synthesis by learning discrete phoneme-level prosodic representations

Abstract

In this paper, we present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset. These features are fed as an input sequence of prosodic labels to a prosody encoder module which augments an autoregressive attention-based text-to-speech model. We utilize various methods in order to improve prosodic control range and coverage, such as augmentation, F0 normalization, balanced clustering for duration and speaker-independent clustering. The final model enables fine-grained phoneme-level prosody control for all speakers contained in the training set, while maintaining the speaker identity. Instead of relying on reference utterances for inference, we introduce a prior prosody encoder which learns the style of each speaker and enables speech synthesis without the requirement of reference audio. We also fine-tune the multispeaker model to unseen speakers with limited amounts of data, as a realistic application scenario and show that the prosody control capabilities are maintained, verifying that the speaker-independent prosodic clustering is effective. Experimental results show that the model has high output speech quality and that the proposed method allows efficient prosody control within each speaker's range despite the variability that a multispeaker setting introduces.

arXiv commentFinal published version available at: Speech Communication. arXiv admin note: substantial text overlap with arXiv:2111.10168

References in corpus (18)

• FastSpeech: Fast, Robust and Controllable Text to Speech
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• MelNet: A Generative Model for Audio in the Frequency Domain
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Rich Prosody Diversity Modelling with Phone-level Mixture Density Network
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
• Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis
• Few Shot Adaptive Normalization Driven Multi-Speaker Speech Synthesis
• Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control