AdaSpeech: Adaptive Text to Speech for Custom Voice

Abstract

Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech data. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions that could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech during training; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. Audio samples are available at https://speechresearch.github.io/adaspeech/.

Accepted by ICLR 2021

References in corpus (7)

• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Deep Speaker: an End-to-End Neural Speaker Embedding System
• Deep Voice: Real-time Neural Text-to-Speech
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Non-Autoregressive Neural Text-to-Speech
• AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

Cited by in corpus (19)

• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• A Survey on Neural Speech Synthesis
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
• Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
• Parameter-Efficient Learning for Text-to-Speech Accent Adaptation
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style
• Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• Hi-Fi Multi-Speaker English TTS Dataset
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
• AdaVocoder: Adaptive Vocoder for Custom Voice
• GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
• Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesis
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech
• AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data