Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language

Abstract

End-to-end text-to-speech synthesis (TTS) can generate highly natural synthetic speech from raw text. However, rendering the correct pitch accents is still a challenging problem for end-to-end TTS. To tackle the challenge of rendering correct pitch accent in Japanese end-to-end TTS, we adopt PnG~BERT, a self-supervised pretrained model in the character and phoneme domain for TTS. We investigate the effects of features captured by PnG~BERT on Japanese TTS by modifying the fine-tuning condition to determine the conditions helpful inferring pitch accents. We manipulate content of PnG~BERT features from being text-oriented to speech-oriented by changing the number of fine-tuned layers during TTS. In addition, we teach PnG~BERT pitch accent information by fine-tuning with tone prediction as an additional downstream task. Our experimental results show that the features of PnG~BERT captured by pretraining contain information helpful inferring pitch accent, and PnG~BERT outperforms baseline Tacotron on accent correctness in a listening test.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
• JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis
• Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection
• Feature reinforcement with word embedding and parsing information in neural TTS
• Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning