CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech

Abstract

Prosody Transfer (PT) is a technique that aims to use the prosody from a source audio as a reference while synthesising speech. Fine-grained PT aims at capturing prosodic aspects like rhythm, emphasis, melody, duration, and loudness, from a source audio at a very granular level and transferring them when synthesising speech in a different target speaker's voice. Current approaches for fine-grained PT suffer from source speaker leakage, where the synthesised speech has the voice identity of the source speaker as opposed to the target speaker. In order to mitigate this issue, they compromise on the quality of PT. In this paper, we propose CopyCat, a novel, many-to-many PT system that is robust to source speaker leakage, without using parallel data. We achieve this through a novel reference encoder architecture capable of capturing temporal prosodic representations which are robust to source speaker leakage. We compare CopyCat against a state-of-the-art fine-grained PT model through various subjective evaluations, where we show a relative improvement of $47\%$ in the quality of prosody transfer and $14\%$ in preserving the target speaker identity, while still maintaining the same naturalness.

References in corpus (8)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Geometric GAN
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization

Cited by in corpus (14)

• A Survey on Neural Speech Synthesis
• Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
• Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis
• Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis
• Emotional Prosody Control for Speech Generation
• Creating New Voices using Normalizing Flows
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations
• On granularity of prosodic representations in expressive text-to-speech
• Word-Level Style Control for Expressive, Non-attentive Speech Synthesis
• Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control
• A Neural TTS System with Parallel Prosody Transfer from Unseen Speakers
• Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis
• Information Sieve: Content Leakage Reduction in End-to-End Prosody For Expressive Speech Synthesis
• Applying the Information Bottleneck Principle to Prosodic Representation Learning