Karaoker: Alignment-free singing voice synthesis with speech training data

Abstract

Existing singing voice synthesis models (SVS) are usually trained on singing data and depend on either error-prone time-alignment and duration features or explicit music score information. In this paper, we propose Karaoker, a multispeaker Tacotron-based model conditioned on voice characteristic features that is trained exclusively on spoken data without requiring time-alignments. Karaoker synthesizes singing voice and transfers style following a multi-dimensional template extracted from a source waveform of an unseen singer/speaker. The model is jointly conditioned with a single deep convolutional encoder on continuous data including pitch, intensity, harmonicity, formants, cepstral peak prominence and octaves. We extend the text-to-speech training objective with feature reconstruction, classification and speaker identification tasks that guide the model to an accurate result. In addition to multitasking, we also employ a Wasserstein GAN training scheme as well as new losses on the acoustic model's output to further refine the quality of the model.

arXiv commentAccepted to INTERSPEECH 2022

References in corpus (5)

• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
• Singing-Tacotron: Global duration control attention and dynamic filter for End-to-end singing voice synthesis
• Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control