Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data
arXiv:2111.07549
Abstract
Recent advancements in end-to-end speech synthesis have made it possible to generate highly natural speech. However, training these models typically requires a large amount of high-fidelity speech data, and for unseen texts, the prosody of synthesized speech is relatively unnatural. To address these issues, we propose to combine a fine-tuned BERT-based front-end with a pre-trained FastSpeech2-based acoustic model to improve prosody modeling. The pre-trained BERT is fine-tuned on the polyphone disambiguation task, the joint Chinese word segmentation (CWS) and part-of-speech (POS) tagging task, and the prosody structure prediction (PSP) task in a multi-task learning framework. FastSpeech 2 is pre-trained on large-scale external data that are noisy but easier to obtain. Experimental results show that both the fine-tuned BERT model and the pre-trained FastSpeech 2 can improve prosody, especially for those structurally complex sentences.
References in corpus (6)
- AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
- Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
- Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models
- Unified Mandarin TTS Front-end Based on Distilled BERT Model
- Probing the phonetic and phonological knowledge of tones in Mandarin TTS models
- PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS