Expressive TTS Training with Frame and Style Reconstruction Loss

Abstract

We propose a novel training strategy for Tacotron-based text-to-speech (TTS) system to improve the expressiveness of speech. One of the key challenges in prosody modeling is the lack of reference that makes explicit modeling difficult. The proposed technique doesn't require prosody annotations from training data. It doesn't attempt to model prosody explicitly either, but rather encodes the association between input text and its prosody styles using a Tacotron-based TTS framework. Our proposed idea marks a departure from the style token paradigm where prosody is explicitly modeled by a bank of prosody embeddings. The proposed training strategy adopts a combination of two objective functions: 1) frame level reconstruction loss, that is calculated between the synthesized and target spectral features; 2) utterance level style reconstruction loss, that is calculated between the deep style features of synthesized and target speech. The proposed style reconstruction loss is formulated as a perceptual loss to ensure that utterance level speech style is taken into consideration during training. Experiments show that the proposed training strategy achieves remarkable performance and outperforms a state-of-the-art baseline in both naturalness and expressiveness. To our best knowledge, this is the first study to incorporate utterance level perceptual quality as a loss function into Tacotron training for improved expressiveness.

Submitted to IEEE/ACM Transactions on Audio, Speech and Language Processing

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0
• Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
• Interactive Text-to-Speech System via Joint Style Analysis
• From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
• An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning

Cited by in corpus (8)

• A Survey on Neural Speech Synthesis
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• GraphSpeech: Syntax-Aware Graph Attention Network For Neural Speech Synthesis
• Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
• Emotional Voice Conversion: Theory, Databases and ESD
• Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS