Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

Abstract

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

References in corpus (6)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Neural Discrete Representation Learning
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Uncovering Latent Style Factors for Expressive Speech Synthesis

Cited by in corpus (37)

• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Sample Efficient Adaptive Text-to-Speech
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• MultiSpeech: Multi-Speaker Text to Speech with Transformer
• Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• Expressive Neural Voice Cloning
• Prosody-controllable spontaneous TTS with neural HMMs
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
• ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
• Multi-Target Emotional Voice Conversion With Neural Vocoders
• Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning
• Controllable neural text-to-speech synthesis using intuitive prosodic features
• MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
• Supervised and Unsupervised Approaches for Controlling Narrow Lexical Focus in Sequence-to-Sequence Speech Synthesis
• Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• On granularity of prosodic representations in expressive text-to-speech
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
• Building a mixed-lingual neural TTS system with only monolingual data
• Towards Fine-Grained Prosody Control for Voice Conversion
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Forward-Backward Decoding for Regularizing End-to-End TTS
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• Pitchtron: Towards audiobook generation from ordinary people's voices
• Towards Natural and Controllable Cross-Lingual Voice Conversion Based on Neural TTS Model and Phonetic Posteriorgram
• Whispered and Lombard Neural Speech Synthesis
• The IQIYI System for Voice Conversion Challenge 2020
• Controllable Emotion Transfer For End-to-End Speech Synthesis
• Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
• Speech BERT Embedding For Improving Prosody in Neural TTS
• An investigation of phrase break prediction in an End-to-End TTS system