Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis

Abstract

Generating versatile and appropriate synthetic speech requires control over the output expression separate from the spoken text. Important non-textual speech variation is seldom annotated, in which case output control must be learned in an unsupervised fashion. In this paper, we perform an in-depth study of methods for unsupervised learning of control in statistical speech synthesis. For example, we show that popular unsupervised training heuristics can be interpreted as variational inference in certain autoencoder models. We additionally connect these models to VQ-VAEs, another, recently-proposed class of deep variational autoencoders, which we show can be derived from a very similar mathematical argument. The implications of these new probabilistic interpretations are discussed. We illustrate the utility of the various approaches with an application to acoustic modelling for emotional speech synthesis, where the unsupervised methods for learning expression control (without access to emotional labels) are found to give results that in many aspects match or surpass the previous best supervised approach.

arXiv comment17 pages, 4 figures

References in corpus (20)

• Auto-Encoding Variational Bayes
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Tutorial on Variational Autoencoders
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Sequential Neural Models with Stochastic Layers
• Inference Suboptimality in Variational Autoencoders
• Amortized Inference Regularization
• VAE with a VampPrior
• ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder
• Uncovering Latent Style Factors for Expressive Speech Synthesis
• Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder
• Fitting New Speakers Based on a Short Untranscribed Sample
• Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder
• Associative Compression Networks for Representation Learning

Cited by in corpus (17)

• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning
• Controllable Data Generation by Deep Learning: A Review
• DiscreTalk: Text-to-Speech as a Machine Translation Problem
• Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0
• VAW-GAN for Singing Voice Conversion with Non-parallel Training Data
• Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis
• Quantization-Based Regularization for Autoencoders
• Learning Speaker Embedding from Text-to-Speech
• Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis
• End-to-End Text-to-Speech using Latent Duration based on VQ-VAE
• Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis