USAT: A Universal Speaker-Adaptive Text-to-Speech Approach

Abstract

Conventional text-to-speech (TTS) research has predominantly focused on enhancing the quality of synthesized speech for speakers in the training dataset. The challenge of synthesizing lifelike speech for unseen, out-of-dataset speakers, especially those with limited reference data, remains a significant and unresolved problem. While zero-shot or few-shot speaker-adaptive TTS approaches have been explored, they have many limitations. Zero-shot approaches tend to suffer from insufficient generalization performance to reproduce the voice of speakers with heavy accents. While few-shot methods can reproduce highly varying accents, they bring a significant storage burden and the risk of overfitting and catastrophic forgetting. In addition, prior approaches only provide either zero-shot or few-shot adaptation, constraining their utility across varied real-world scenarios with different demands. Besides, most current evaluations of speaker-adaptive TTS are conducted only on datasets of native speakers, inadvertently neglecting a vast portion of non-native speakers with diverse accents. Our proposed framework unifies both zero-shot and few-shot speaker adaptation strategies, which we term as "instant" and "fine-grained" adaptations based on their merits. To alleviate the insufficient generalization performance observed in zero-shot speaker adaptation, we designed two innovative discriminators and introduced a memory mechanism for the speech decoder. To prevent catastrophic forgetting and reduce storage implications for few-shot speaker adaptation, we designed two adapters and a unique adaptation procedure.

15 pages, 13 figures. Copyright has been transferred to IEEE

References in corpus (11)

• Res2Net: A New Multi-scale Backbone Architecture
• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• Attentive Statistics Pooling for Deep Speaker Embedding
• A Survey on Neural Speech Synthesis
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
• Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation
• How Generalizable are Deepfake Image Detectors? An Empirical Study