Code-Mixed Text to Speech Synthesis under Low-Resource Constraints

Abstract

Text-to-speech (TTS) systems are an important component in voice-based e-commerce applications. These applications include end-to-end voice assistant and customer experience (CX) voice bot. Code-mixed TTS is also relevant in these applications since the product names are commonly described in English while the surrounding text is in a regional language. In this work, we describe our approaches for production quality code-mixed Hindi-English TTS systems built for e-commerce applications. We propose a data-oriented approach by utilizing monolingual data sets in individual languages. We leverage a transliteration model to convert the Roman text into a common Devanagari script and then combine both datasets for training. We show that such single script bi-lingual training without any code-mixing works well for pure code-mixed test sets. We further present an exhaustive evaluation of single-speaker adaptation and multi-speaker training with Tacotron2 + Waveglow setup to show that the former approach works better. These approaches are also coupled with transfer learning and decoder-only fine-tuning to improve performance. We compare these approaches with the Google TTS and report a positive CMOS score of 0.02 with the proposed transfer learning approach. We also perform low-resource voice adaptation experiments to show that a new voice can be onboarded with just 3 hrs of data. This highlights the importance of our pre-trained models in resource-constrained settings. This subjective evaluation is performed on a large number of out-of-domain pure code-mixed sentences to demonstrate the high quality of the systems.

arXiv commentAccepted at SPECOM 2023

References in corpus (11)

• Attention-Based Models for Speech Recognition
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• A Survey on Neural Speech Synthesis
• Neural Voice Cloning with a Few Samples
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models
• Attention based end to end Speech Recognition for Voice Search in Hindi and English