Many-to-Many Voice Transformer Network

Abstract

This paper proposes a voice conversion (VC) method based on a sequence-to-sequence (S2S) learning framework, which enables simultaneous conversion of the voice characteristics, pitch contour, and duration of input speech. We previously proposed an S2S-based VC method using a transformer network architecture called the voice transformer network (VTN). The original VTN was designed to learn only a mapping of speech feature sequences from one speaker to another. The main idea we propose is an extension of the original VTN that can simultaneously learn mappings among multiple speakers. This extension called the many-to-many VTN makes it able to fully use available training data collected from multiple speakers by capturing common latent features that can be shared across different speakers. It also allows us to introduce a training loss called the identity mapping loss to ensure that the input feature sequence will remain unchanged when the source and target speaker indices are the same. Using this particular loss for model training has been found to be extremely effective in improving the performance of the model at test time. We conducted speaker identity conversion experiments and found that our model obtained higher sound quality and speaker similarity than baseline methods. We also found that our model, with a slight modification to its architecture, could handle any-to-many conversion tasks reasonably well.

arXiv commentsubmitted to IEEE/ACM Trans. ASLP. Please also refer to our related article: arXiv:1811.01609

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• Attention-Based Models for Speech Recognition
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
• On Layer Normalization in the Transformer Architecture
• Efficient Neural Audio Synthesis
• The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• FloWaveNet : A Generative Flow for Raw Audio
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation

Cited by in corpus (5)

• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion
• Non-autoregressive sequence-to-sequence voice conversion
• Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks