An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
arXiv:2008.03648
Abstract
Speaker identity is one of the important characteristics of human speech. In voice conversion, we change the speaker identity from one to another, while keeping the linguistic content unchanged. Voice conversion involves multiple speech processing techniques, such as speech analysis, spectral conversion, prosody conversion, speaker characterization, and vocoding. With the recent advances in theory and practice, we are now able to produce human-like voice quality with high speaker similarity. In this paper, we provide a comprehensive overview of the state-of-the-art of voice conversion techniques and their performance evaluation methods from the statistical approaches to deep learning, and discuss their promise and limitations. We will also report the recent Voice Conversion Challenges (VCC), the performance of the current state of technology, and provide a summary of the available resources for voice conversion research.
arXiv commentaccepted by IEEE/ACM Transactions on Audio, Speech and Language Processing
References in corpus (42)
- LSTM: A Search Space Odyssey
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- WaveNet: A Generative Model for Raw Audio
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Neural Discrete Representation Learning
- Convolutional Sequence to Sequence Learning
- Self-Attention Generative Adversarial Networks
- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
- Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization
- Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
- MOSNet: Deep Learning based Objective Assessment for Voice Conversion
- Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
- Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
- Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
- Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
- Evaluating Voice Conversion-based Privacy Protection against Informed Attackers
- Efficient Neural Audio Synthesis
- Cycle-Consistent Speech Enhancement
- AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech
- Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoders
- StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
- Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining
- Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision
- Wavelet speech enhancement based on nonnegative matrix factorization
- Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
- Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model based on BLSTM
- Refined WaveNet Vocoder for Variational Autoencoder Based Voice Conversion
- Expressive TTS Training with Frame and Style Reconstruction Loss
- Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
- Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data
- VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
- NAUTILUS: a Versatile Voice Cloning System
- Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion
- Voice Conversion Using Sequence-to-Sequence Learning of Context Posterior Probabilities
- Multi-Target Emotional Voice Conversion With Neural Vocoders
- Quasi-Periodic WaveNet Vocoder: A Pitch Dependent Dilated Convolution Model for Parametric Speech Generation
- Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet
- Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis
- Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling
- WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss
- Teacher-Student Training for Robust Tacotron-based TTS
- Transferring Source Style in Non-Parallel Voice Conversion
Cited by in corpus (7)
- Expressive TTS Training with Frame and Style Reconstruction Loss
- VAW-GAN for Singing Voice Conversion with Non-parallel Training Data
- Many-to-Many Voice Transformer Network
- Emotional Voice Conversion: Theory, Databases and ESD
- FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion
- FastVC: Fast Voice Conversion with non-parallel data
- Spectrum and Prosody Conversion for Cross-lingual Voice Conversion with CycleGAN