SQuId: Measuring Speech Naturalness in Many Languages

Abstract

Much of text-to-speech research relies on human evaluation, which incurs heavy costs and slows down the development process. The problem is particularly acute in heavily multilingual applications, where recruiting and polling judges can take weeks. We introduce SQuId (Speech Quality Identification), a multilingual naturalness prediction model trained on over a million ratings and tested in 65 locales-the largest effort of this type to date. The main insight is that training one model on many locales consistently outperforms mono-locale baselines. We present our task, the model, and show that it outperforms a competitive baseline based on w2v-BERT and VoiceMOS by 50.0%. We then demonstrate the effectiveness of cross-locale transfer during fine-tuning and highlight its effect on zero-shot locales, i.e., locales for which there is no fine-tuning data. Through a series of analyses, we highlight the role of non-linguistic effects such as sound artifacts in cross-locale transfer. Finally, we present the effect of our design decision, e.g., model size, pre-training diversity, and language rebalancing with several ablation experiments.

Accepted at ICASSP 2023, with additional material in the appendix

References in corpus (7)

• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• mSLAM: Massively multilingual joint pre-training for speech and text
• AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech
• Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis
• How Phonotactics Affect Multilingual and Zero-shot ASR Performance