Hierarchical Generative Modeling for Controllable Speech Synthesis

Abstract

This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.

arXiv comment27 pages, accepted to ICLR 2019

References in corpus (10)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Deep Voice: Real-time Neural Text-to-Speech
• Neural Voice Cloning with a Few Samples
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Fitting New Speakers Based on a Short Untranscribed Sample
• Listening while Speaking: Speech Chain by Deep Learning

Cited by in corpus (24)

• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• MelNet: A Generative Model for Audio in the Frequency Domain
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• MultiSpeech: Multi-Speaker Text to Speech with Transformer
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• GraphSpeech: Syntax-Aware Graph Attention Network For Neural Speech Synthesis
• Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
• DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modeling
• A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Injecting Text in Self-Supervised Speech Pretraining
• Pitchtron: Towards audiobook generation from ordinary people's voices
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data
• Neural voice cloning with a few low-quality samples
• Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
• Enhancing audio quality for expressive Neural Text-to-Speech