Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis

Abstract

This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer level representations on coarser level ones. Additionally, it imposes hierarchical conditioning across all latent dimensions using a conditional variational auto-encoder (VAE) with an auto-regressive structure. Evaluation of reconstruction performance illustrates that the new structure does not degrade the model while allowing better interpretability. Interpretations of prosody attributes are provided together with the comparison between word-level and phone-level prosody representations. Moreover, both qualitative and quantitative evaluations are used to demonstrate the improvement in the disentanglement of the latent dimensions.

to appear in ICASSP 2020

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling factors of variation in deep representations using adversarial training
• Structured Disentangled Representations
• RNADE: The real-valued neural autoregressive density-estimator
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis

Cited by in corpus (4)

• Parallel Tacotron: Non-Autoregressive and Controllable TTS
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
• Exploiting Deep Sentential Context for Expressive End-to-End Speech Synthesis