Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice

Abstract

Neural TTS has shown it can generate high quality synthesized speech. In this paper, we investigate the multi-speaker latent space to improve neural TTS for adapting the system to new speakers with only several minutes of speech or enhancing a premium voice by utilizing the data from other speakers for richer contextual coverage and better generalization. A multi-speaker neural TTS model is built with the embedded speaker information in both spectral and speaker latent space. The experimental results show that, with less than 5 minutes of training data from a new speaker, the new model can achieve an MOS score of 4.16 in naturalness and 4.64 in speaker similarity close to human recordings (4.74). For a well-trained premium voice, we can achieve an MOS score of 4.5 for out-of-domain texts, which is comparable to an MOS of 4.58 for professional recordings, and significantly outperforms single speaker result of 4.28.

References in corpus (5)

• WaveNet: A Generative Model for Raw Audio
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Deep Speaker: an End-to-End Neural Speaker Embedding System
• Neural Voice Cloning with a Few Samples
• Fitting New Speakers Based on a Short Untranscribed Sample

Cited by in corpus (10)

• Improving Readability for Automatic Speech Recognition Transcription
• Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings
• A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
• Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
• Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages
• On Addressing Practical Challenges for RNN-Transducer
• AI based Presentation Creator With Customized Audio Content Delivery
• Combining speakers of multiple languages to improve quality of neural voices
• Speech BERT Embedding For Improving Prosody in Neural TTS