Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech

Abstract

For personalized speech generation, a neural text-to-speech (TTS) model must be successfully implemented with limited data from a target speaker. To this end, the baseline TTS model needs to be amply generalized to out-of-domain data (i.e., target speaker's speech). However, approaches to address this out-of-domain generalization problem in TTS have yet to be thoroughly studied. In this work, we propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. In particular, we prune off redundant connections from self-attention layers whose attention weights are below the threshold. To flexibly determine the pruning strength for searching optimal degree of generalization, we also propose a new differentiable pruning method that allows the model to automatically learn the thresholds. Evaluations on zero-shot multi-speaker TTS verify the effectiveness of our method in terms of voice quality and speaker similarity.

arXiv commentINTERSPEECH 2023

References in corpus (14)

• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• SpeechBrain: A General-Purpose Speech Toolkit
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• Sparse Sinkhorn Attention
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus
• Learned Token Pruning for Transformers
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• Generalizing RNN-Transducer to Out-Domain Audio via Sparse Self-Attention Layers