SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech

Abstract

Zero-shot multi-speaker text-to-speech (ZSM-TTS) models aim to generate a speech sample with the voice characteristic of an unseen speaker. The main challenge of ZSM-TTS is to increase the overall speaker similarity for unseen speakers. One of the most successful speaker conditioning methods for flow-based multi-speaker text-to-speech (TTS) models is to utilize the functions which predict the scale and bias parameters of the affine coupling layers according to the given speaker embedding vector. In this letter, we improve on the previous speaker conditioning method by introducing a speaker-normalized affine coupling (SNAC) layer which allows for unseen speaker speech synthesis in a zero-shot manner leveraging a normalization-based conditioning technique. The newly designed coupling layer explicitly normalizes the input by the parameters predicted from a speaker embedding vector while training, enabling an inverse process of denormalizing for a new speaker embedding at inference. The proposed conditioning scheme yields the state-of-the-art performance in terms of the speech quality and speaker similarity in a ZSM-TTS setting.

arXiv commentAccepted to IEEE Signal Processing Letters

References in corpus (20)

• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• WaveNet: A Generative Model for Raw Audio
• Instance Normalization: The Missing Ingredient for Fast Stylization
• NICE: Non-linear Independent Components Estimation
• FastSpeech: Fast, Robust and Controllable Text to Speech
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• SpeechBrain: A General-Purpose Speech Toolkit
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Deep Voice: Real-time Neural Text-to-Speech
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Neural Voice Cloning with a Few Samples
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation
• Semi-Conditional Normalizing Flows for Semi-Supervised Learning