Hierarchical Sequence to Sequence Voice Conversion with Limited Data

Abstract

We present a voice conversion solution using recurrent sequence to sequence modeling for DNNs. Our solution takes advantage of recent advances in attention based modeling in the fields of Neural Machine Translation (NMT), Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). The problem consists of converting between voices in a parallel setting when {\it $<$source,target$>$} audio pairs are available. Our seq2seq architecture makes use of a hierarchical encoder to summarize input audio frames. On the decoder side, we use an attention based architecture used in recent TTS works. Since there is a dearth of large multispeaker voice conversion databases needed for training DNNs, we resort to training the network with a large single speaker dataset as an autoencoder. This is then adapted for the smaller multispeaker voice conversion datasets available for voice conversion. In contrast with other voice conversion works that use $F_0$, duration and linguistic features, our system uses mel spectrograms as the audio representation. Output mel frames are converted back to audio using a wavenet vocoder.

References in corpus (21)

• WaveNet: A Generative Model for Raw Audio
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Autoencoding beyond pixels using a learned similarity metric
• Deep Voice: Real-time Neural Text-to-Speech
• Neural Machine Translation in Linear Time
• Listen, Attend and Spell
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Neural Voice Cloning with a Few Samples
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• Improving Variational Inference with Inverse Autoregressive Flow
• Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
• Tacotron: Towards End-to-End Speech Synthesis
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
• Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision
• Video Pixel Networks
• ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion
• Fitting New Speakers Based on a Short Untranscribed Sample
• AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms

Cited by in corpus (2)

• Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion