MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
arXiv:2203.00931 · doi:10.1109/TASLP.2023.3294712
Abstract
We propose a multi-singer emotional singing voice synthesizer, Muse-SVS, that expresses emotion at various intensity levels by controlling subtle changes in pitch, energy, and phoneme duration while accurately following the score. To control multiple style attributes while avoiding loss of fidelity and expressiveness due to interference between attributes, Muse-SVS represents all attributes and their relations together by a joint embedding in a unified embedding space. Muse-SVS can express emotional intensity levels not included in the training data through embedding interpolation and extrapolation. We also propose a statistical pitch predictor to express pitch variance according to emotional intensity, and a context-aware residual duration predictor to prevent the accumulation of variances in phoneme duration, which is crucial for synchronization with instrumental parts. In addition, we propose a novel ASPP-Transformer, which combines atrous spatial pyramid pooling (ASPP) and Transformer, to improve fidelity and expressiveness by referring to broad contexts. In experiments, Muse-SVS exhibited improved fidelity, expressiveness, and synchronization performance compared with baseline models. The visualization results show that Muse-SVS effectively express the variance in pitch, energy, and phoneme duration according to emotional intensity. To the best of our knowledge, Muse-SVS is the first neural SVS capable of controlling emotional intensity.
arXiv comment13 pages, 11 figures
References in corpus (14)
- FastSpeech: Fast, Robust and Controllable Text to Speech
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
- Deep Voice 2: Multi-Speaker Neural Text-to-Speech
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
- WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN
- CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
- Uncovering Latent Style Factors for Expressive Speech Synthesis
- HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
- Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System
- MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder
- Expressive Singing Synthesis Using Local Style Token and Dual-path Pitch Encoder
- UniTTS: Residual Learning of Unified Embedding Space for Speech Style Control