NAUTILUS: a Versatile Voice Cloning System

Abstract

We introduce a novel speech synthesis system, called NAUTILUS, that can generate speech with a target voice either from a text input or a reference utterance of an arbitrary source speaker. By using a multi-speaker speech corpus to train all requisite encoders and decoders in the initial training stage, our system can clone unseen voices using untranscribed speech of target speakers on the basis of the backpropagation algorithm. Moreover, depending on the data circumstance of the target speaker, the cloning strategy can be adjusted to take advantage of additional data and modify the behaviors of text-to-speech (TTS) and/or voice conversion (VC) systems to accommodate the situation. We test the performance of the proposed framework by using deep convolution layers to model the encoders, decoders and WaveNet vocoder. Evaluations show that it achieves comparable quality with state-of-the-art TTS and VC systems when cloning with just five minutes of untranscribed speech. Moreover, it is demonstrated that the proposed framework has the ability to switch between TTS and VC with high speaker consistency, which will be useful for many applications.

arXiv commentSubmitted to The IEEE/ACM Transactions on Audio, Speech, and Language Processing

References in corpus (11)

• WaveNet: A Generative Model for Raw Audio
• Autoencoding beyond pixels using a learned similarity metric
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Quasi-Recurrent Neural Networks
• Sample Efficient Adaptive Text-to-Speech
• Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining
• Refined WaveNet Vocoder for Variational Autoencoder Based Voice Conversion
• A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation
• Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings
• Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech

Cited by in corpus (3)

• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion
• AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data