AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
arXiv:1905.05879
Abstract
Non-parallel many-to-many voice conversion, as well as zero-shot voice conversion, remain under-explored areas. Deep style transfer algorithms, such as generative adversarial networks (GAN) and conditional variational autoencoder (CVAE), are being applied as new solutions in this field. However, GAN training is sophisticated and difficult, and there is no strong evidence that its generated speech is of good perceptual quality. On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN. In this paper, we propose a new style transfer scheme that involves only an autoencoder with a carefully designed bottleneck. We formally show that this scheme can achieve distribution-matching style transfer by training only on a self-reconstruction loss. Based on this scheme, we proposed AUTOVC, which achieves state-of-the-art results in many-to-many voice conversion with non-parallel data, and which is the first to perform zero-shot voice conversion.
arXiv commentTo Appear in Thirty-sixth International Conference on Machine Learning (ICML 2019)
References in corpus (12)
- WaveNet: A Generative Model for Raw Audio
- VoxCeleb: a large-scale speaker identification dataset
- Semi-Supervised Learning with Deep Generative Models
- Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
- Deep Voice: Real-time Neural Text-to-Speech
- Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
- Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
- Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoders
- ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder
- StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
- Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder
- Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
Cited by in corpus (39)
- MakeItTalk: Speaker-Aware Talking-Head Animation
- A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion
- F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder
- Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling
- CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
- Unsupervised Speech Decomposition via Triple Information Bottleneck
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
- Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglement
- Towards Robust Neural Vocoding for Speech Generation: A Survey
- WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
- TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training
- VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
- Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification
- VSMask: Defending Against Voice Synthesis Attack via Real-Time Predictive Perturbation
- Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion
- End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions
- ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed
- Creating New Voices using Normalizing Flows
- Many-to-Many Voice Transformer Network
- Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE
- SpeechNet: A Universal Modularized Model for Speech Processing Tasks
- Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion
- Parallel Synthesis for Autoregressive Speech Generation
- StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts
- StarGAN-VC+ASR: StarGAN-based Non-Parallel Voice Conversion Regularized by Automatic Speech Recognition
- ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
- Defending Your Voice: Adversarial Attack on Voice Conversion
- FastVC: Fast Voice Conversion with non-parallel data
- Unsupervised Audiovisual Synthesis via Exemplar Autoencoders
- Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
- S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations
- Adversarially learning disentangled speech representations for robust multi-factor voice conversion
- CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion
- STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
- Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
- Speech Enhancement-assisted Voice Conversion in Noisy Environments
- An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentation
- How Far Are We from Robust Voice Conversion: A Survey
- Zero-shot Singing Technique Conversion