Parameter-Efficient Learning for Text-to-Speech Accent Adaptation

Abstract

This paper presents a parameter-efficient learning (PEL) to develop a low-resource accent adaptation for text-to-speech (TTS). A resource-efficient adaptation from a frozen pre-trained TTS model is developed by using only 1.2\% to 0.8\% of original trainable parameters to achieve competitive performance in voice synthesis. Motivated by a theoretical foundation of optimal transport (OT), this study carries out PEL for TTS where an auxiliary unsupervised loss based on OT is introduced to maximize a difference between the pre-trained source domain and the (unseen) target domain, in addition to its supervised training loss. Further, we leverage upon this unsupervised loss refinement to boost system performance via either sliced Wasserstein distance or maximum mean discrepancy. The merit of this work is demonstrated by fulfilling PEL solutions based on residual adapter learning, and model reprogramming when evaluating the Mandarin accent adaptation. Experiment results show that the proposed methods can achieve competitive naturalness with parameter-efficient decoder fine-tuning, and the auxiliary unsupervised loss improves model performance empirically.

Accepted to Interspeech 2023

References in corpus (10)

• On the Opportunities and Risks of Foundation Models
• Towards a Unified View of Parameter-Efficient Transfer Learning
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition
• Offsite-Tuning: Transfer Learning without Full Model
• Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation

Cited by in corpus (1)

• Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition