Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining

Abstract

We introduce a novel sequence-to-sequence (seq2seq) voice conversion (VC) model based on the Transformer architecture with text-to-speech (TTS) pretraining. Seq2seq VC models are attractive owing to their ability to convert prosody. While seq2seq models based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have been successfully applied to VC, the use of the Transformer network, which has shown promising results in various speech processing tasks, has not yet been investigated. Nonetheless, their data-hungry property and the mispronunciation of converted speech make seq2seq models far from practical. To this end, we propose a simple yet effective pretraining technique to transfer knowledge from learned TTS models, which benefit from large-scale, easily accessible TTS corpora. VC models initialized with such pretrained model parameters are able to generate effective hidden representations for high-fidelity, highly intelligible converted speech. Experimental results show that such a pretraining scheme can facilitate data-efficient training and outperform an RNN-based seq2seq VC model in terms of intelligibility, naturalness, and similarity.

Preprint. Work in progress

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision
• ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data

Cited by in corpus (15)

• ESPnet2-TTS: Extending the Edge of TTS Research
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• NAUTILUS: a Versatile Voice Cloning System
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• Emotional Voice Conversion: Theory, Databases and ESD
• ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
• Transfer Learning from Monolingual ASR to Transcription-free Cross-lingual Voice Conversion
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion
• Axial Residual Networks for CycleGAN-based Voice Conversion
• crank: An Open-Source Software for Nonparallel Voice Conversion Based on Vector-Quantized Variational Autoencoder
• MASS: Multi-task Anthropomorphic Speech Synthesis Framework
• Non-autoregressive sequence-to-sequence voice conversion
• Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
• Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations
• EMA2S: An End-to-End Multimodal Articulatory-to-Speech System