Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
arXiv:1911.00137
Abstract
We have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo, a performer plays multiple characters, and conversations or dialogues between the characters make the story progress. To investigate how close the quality of synthesized rakugo speech can approach that of professionals' speech, we modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis system that can produce speech that sounds as natural as human speech albeit under limited conditions, and an enhanced version of it with self-attention to better consider long-term dependencies. We also used global style tokens and manually labeled context features to enrich speaking styles. Through a listening test, we measured not only naturalness but also distinguishability of characters, understandability of the content, and the degree of entertainment. Although we found that the speech synthesis models could not yet reach the professional level, the results of the listening test provided interesting insights: 1) we should not focus only on the naturalness of synthesized speech but also the distinguishability of characters and the understandability of the content to further entertain audiences; 2) the fundamental frequency (fo) expressions of synthesized speech are poorer than those of human speech, and more entertaining speech should have richer fo expression. Although there is room for improvement, we believe this is an important stepping stone toward achieving entertaining speech synthesis at the professional level.
arXiv commentResubmitted to IEEE Access
References in corpus (14)
- Neural Machine Translation by Jointly Learning to Align and Translate
- WaveNet: A Generative Model for Raw Audio
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- A guide to convolution arithmetic for deep learning
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
- Long Short-Term Memory-Networks for Machine Reading
- Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- MelNet: A Generative Model for Audio in the Frequency Domain
- Uncovering Latent Style Factors for Expressive Speech Synthesis
- Hierarchical Generative Modeling for Controllable Speech Synthesis
- A comparison of recent waveform generation and acoustic modeling methods for neural-network-based speech synthesis
- Investigating accuracy of pitch-accent annotations in neural network-based speech synthesis and denoising effects