Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

Abstract

We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

arXiv commentNeurIPS 2018

References in corpus (8)

• WaveNet: A Generative Model for Raw Audio
• VoxCeleb2: Deep Speaker Recognition
• VoxCeleb: a large-scale speaker identification dataset
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Neural Voice Cloning with a Few Samples
• Sample Efficient Adaptive Text-to-Speech
• Fitting New Speakers Based on a Short Untranscribed Sample

Cited by in corpus (54)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• TweepFake: about Detecting Deepfake Tweets
• Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
• Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder
• Sample Efficient Adaptive Text-to-Speech
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• A Novel GDP Prediction Technique based on Transfer Learning using CO2 Emission Dataset
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
• Attack Agnostic Dataset: Towards Generalization and Stabilization of Audio DeepFake Detection
• Efficient Personalized Speech Enhancement through Self-Supervised Learning
• FloWaveNet : A Generative Flow for Raw Audio
• Learning pronunciation from a foreign language in speech synthesis networks
• FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
• Defend Data Poisoning Attacks on Voice Authentication
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
• Heterogeneous Target Speech Separation
• SANE-TTS: Stable And Natural End-to-End Multilingual Text-to-Speech
• VSMask: Defending Against Voice Synthesis Attack via Real-Time Predictive Perturbation
• NAUTILUS: a Versatile Voice Cloning System
• Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild
• Giving Robots a Voice: Human-in-the-Loop Voice Creation and open-ended Labeling
• Emotional Prosody Control for Speech Generation
• Towards Unconstrained Audio Splicing Detection and Localization with Neural Networks
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• Introducing Model Inversion Attacks on Automatic Speaker Recognition
• Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE
• Hi-Fi Multi-Speaker English TTS Dataset
• Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech
• ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
• Fine-grained Noise Control for Multispeaker Speech Synthesis
• CUHK-EE Voice Cloning System for ICASSP 2021 M2VoC Challenge
• Speaking style adaptation in Text-To-Speech synthesis using Sequence-to-sequence models with attention
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation
• DiDiSpeech: A Large Scale Mandarin Speech Corpus
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints
• Residual Information in Deep Speaker Embedding Architectures
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Development of Automatic Speech Recognition for Kazakh Language using Transfer Learning
• Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement
• Deepfake audio as a data augmentation technique for training automatic speech to text transcription models
• Continual Speaker Adaptation for Text-to-Speech Synthesis
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration
• Large-scale Speaker Retrieval on Random Speaker Variability Subspace
• TranSTYLer: Multimodal Behavioral Style Transfer for Facial and Body Gestures Generation
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
• BarrierBypass: Out-of-Sight Clean Voice Command Injection Attacks through Physical Barriers
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech
• Zero-shot Singing Technique Conversion