Neural Voice Cloning with a Few Samples

Abstract

Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

References in corpus (13)

• WaveNet: A Generative Model for Raw Audio
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Conditional Image Generation with PixelCNN Decoders
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Exploring the Limits of Language Modeling
• Regularizing and Optimizing LSTM Language Models
• Deep Speaker: an End-to-End Neural Speaker Embedding System
• Deep Voice: Real-time Neural Text-to-Speech
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
• Embedding-Based Speaker Adaptive Training of Deep Neural Networks

Cited by in corpus (22)

• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• Sample Efficient Adaptive Text-to-Speech
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus
• FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
• Review of end-to-end speech synthesis technology based on deep learning
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• A Plug-in Method for Representation Factorization in Connectionist Models
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• Speaking style adaptation in Text-To-Speech synthesis using Sequence-to-sequence models with attention
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
• The AS-NU System for the M2VoC Challenge
• In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data
• AdaVocoder: Adaptive Vocoder for Custom Voice
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Interactive Storytelling for Children: A Case-study of Design and Development Considerations for Ethical Conversational AI
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Facial Keypoint Sequence Generation from Audio
• Learning to Compensate: A Deep Neural Network Framework for 5G Power Amplifier Compensation