MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
arXiv:2203.00931 · doi:10.1109/TASLP.2023.3294712
Abstract
We propose a multi-singer emotional singing voice synthesizer, Muse-SVS, that expresses emotion at various intensity levels by controlling subtle changes in pitch, energy, and phoneme duration while accurately following the score. To control multiple style attributes while avoiding loss of fidelity and expressiveness due to interference between attributes, Muse-SVS represents all attributes and their relations together by a joint embedding in a unified embedding space. Muse-SVS can express emotional intensity levels not included in the training data through embedding interpolation and extrapolation. We also propose a statistical pitch predictor to express pitch variance according to emotional intensity, and a context-aware residual duration predictor to prevent the accumulation of variances in phoneme duration, which is crucial for synchronization with instrumental parts. In addition, we propose a novel ASPP-Transformer, which combines atrous spatial pyramid pooling (ASPP) and Transformer, to improve fidelity and expressiveness by referring to broad contexts. In experiments, Muse-SVS exhibited improved fidelity, expressiveness, and synchronization performance compared with baseline models. The visualization results show that Muse-SVS effectively express the variance in pitch, energy, and phoneme duration according to emotional intensity. To the best of our knowledge, Muse-SVS is the first neural SVS capable of controlling emotional intensity.
13 pages, 11 figures
References in corpus (7)
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
- CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
- Uncovering Latent Style Factors for Expressive Speech Synthesis
- HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
- Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System
- MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder
- Expressive Singing Synthesis Using Local Style Token and Dual-path Pitch Encoder