Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement

Abstract

This paper presents a novel design of neural network system for fine-grained style modeling, transfer and prediction in expressive text-to-speech (TTS) synthesis. Fine-grained modeling is realized by extracting style embeddings from the mel-spectrograms of phone-level speech segments. Collaborative learning and adversarial learning strategies are applied in order to achieve effective disentanglement of content and style factors in speech and alleviate the "content leakage" problem in style modeling. The proposed system can be used for varying-content speech style transfer in the single-speaker scenario. The results of objective and subjective evaluation show that our system performs better than other fine-grained speech style transfer models, especially in the aspect of content preservation. By incorporating a style predictor, the proposed system can also be used for text-to-speech synthesis. Audio samples are provided for system demonstration https://daxintan-cuhk.github.io/pl-csd-speech .

arXiv commentAccepted by Interspeech 2021

References in corpus (16)

• FastSpeech: Fast, Robust and Controllable Text to Speech
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Tacotron: Towards End-to-End Speech Synthesis
• DurIAN: Duration Informed Attention Network For Multimodal Synthesis
• ESPnet: End-to-End Speech Processing Toolkit
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Expressive TTS Training with Frame and Style Reconstruction Loss
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion

Cited by in corpus (6)

• Emotion Intensity and its Control for Emotional Voice Conversion
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• Towards Multi-Scale Style Control for Expressive Speech Synthesis
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• Applying the Information Bottleneck Principle to Prosodic Representation Learning