Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus

Abstract

Training a text-to-speech (TTS) model requires a large scale text labeled speech corpus, which is troublesome to collect. In this paper, we propose a transfer learning framework for TTS that utilizes a large amount of unlabeled speech dataset for pre-training. By leveraging wav2vec2.0 representation, unlabeled speech can highly improve performance, especially in the lack of labeled speech. We also extend the proposed method to zero-shot multi-speaker TTS (ZS-TTS). The experimental results verify the effectiveness of the proposed method in terms of naturalness, intelligibility, and speaker generalization. We highlight that the single speaker TTS model fine-tuned on the only 10 minutes of labeled dataset outperforms the other baselines, and the ZS-TTS model fine-tuned on the only 30 minutes of single speaker dataset can generate the voice of the arbitrary speaker, by pre-training on unlabeled multi-speaker speech corpus.

arXiv commentAccepted by Interspeech2022

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Representation Learning with Contrastive Predictive Coding
• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• SpeechBrain: A General-Purpose Speech Toolkit
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Neural Voice Cloning with a Few Samples
• Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

Cited by in corpus (2)

• USAT: A Universal Speaker-Adaptive Text-to-Speech Approach
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech